Vehicle exterior environment recognition device

ABSTRACT

A vehicle exterior environment recognition device includes an image acquiring module that acquires an image, a traffic sign identifying module that identifies a circle of a predetermined radius centering on any one of pixels in the image as a traffic sign, a traffic sign content recognizing module that recognizes content of the identified traffic sign, and a traffic sign content determining module that uses at least one template for one certain country to integrate traffic sign integration points based on correlation evaluation values with the content of the recognized traffic sign, uses a template for each of a plurality of countries corresponding to the content of the traffic sign having the traffic sign integration points to integrate total points by country based on overall evaluation values of the content of the recognized traffic sign, and conclusively determines a currently-traveling country.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority from Japanese Patent ApplicationNo. 2014-070487 filed on Mar. 28, 2014, the entire content of which arehereby incorporated by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to a vehicle exterior environmentrecognition device that recognizes content of a traffic sign that isconfirmed outside a vehicle.

2. Related Art

Conventionally, there are techniques to detect a particular object, suchas another vehicle located ahead of a vehicle (for example, see JapanesePatent No. 3349060B). Such a technique is used to avoid a collision witha leading vehicle (collision avoidance control), or to control adistance between the two vehicles to be maintained at a safe distance(cruise control). In order to reduce accidents resulting from speeding,demands for techniques to recognize a speed limit provided for each roadand control the speed of the vehicle is increasing.

In order to safely travel the vehicle within a speed limit, it isnecessary to recognize content of a traffic sign located at a roadshoulder or a gate, and to correctly grasp the speed limit of thecurrently traveling road. For example, Japanese Unexamined PatentApplication Publication (JP-A) No. 2012-243051 discloses a technique toapply Hough transform to part corresponding to an edge on a screenimage, and recognize an image of a circular traffic sign (hereinafter,an image of the traffic sign is also simply referred to as “the trafficsign”). In this technique, a processing load required for the Houghtransform is reduced, whereby the efficiency of identifying the trafficsign can be improved.

In order to perform the Hough transform and to recognize the circulartraffic sign, a feature point corresponding to part of the circumferenceof the circular traffic sign is first identified, points on thecircumference that are separated from the feature point by apredetermined distance are voted, and a candidate of the traffic signhaving a center position and a radius of the circle is identifiedaccording to the number of votes obtained. Then, the content of thetraffic sign is recognized by applying, for example, pattern matching,to the candidate of the traffic sign. However, the traffic signs thatpresent a speed limit may be different in the size and/or the shape ofnumerals indicating the speed limits, and/or in the distance between thenumerals, in each country. Therefore, if the pattern matching is simplyperformed disregarding the differences, the matching may be establishedfor the content that should not be originally matched, whereby troublesmay be caused in the safe traveling.

SUMMARY OF THE INVENTION

The present disclosure is made in view of the above situations, and apurpose of the present disclosure is to provide a vehicle exteriorenvironment recognition device that can improve identification accuracyof content of a traffic sign by appropriately determining acurrently-traveling country, while reducing a processing load.

An aspect of the present disclosure provides a vehicle exteriorenvironment recognition device including: an image acquiring module thatacquires an image; a traffic sign identifying module that identifies acircle of a predetermined radius centering on any one of pixels in theimage as a traffic sign; a traffic sign content recognizing module thatrecognizes content of the identified traffic sign; and a traffic signcontent determining module that uses at least one template for onecertain country to integrate traffic sign integration points based oncorrelation evaluation values with the recognized content of theidentified traffic sign, uses a template for each of a plurality ofcountries corresponding to the content of the traffic sign having thetraffic sign integration points to integrate total points by countrybased on overall evaluation values of the content of the recognizedtraffic sign, and conclusively determines a currently-traveling country.

The traffic sign content determining module may apply weighting to theoverall evaluation values a currently-recognized country and a countryadjacent to the currently-recognized country so that thecurrently-recognized country and the country adjacent to thecurrently-recognized country are easily selected.

The traffic sign content determining module may store images of trafficsigns having the traffic sign integration points in an image memory,integrate the total points by country based on correlation evaluationvalues of the content of the recognized traffic sign, and execute theprocessing that conclusively determines the currently-traveling country,during an idle time of the processing that integrates the traffic signintegration points based on the correlation evaluation values of thecontent of the recognized traffic sign using the at least one templateof the one certain country.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and not by wayof limitation in the figures of the accompanying drawings, in which thelike reference numerals indicate like elements and in which:

FIG. 1 is a block diagram illustrating a connecting relation of avehicle exterior environment recognition system;

FIGS. 2A and 2B are diagrams illustrating a color image and a distanceimage, respectively;

FIG. 3 is a functional block diagram schematically illustratingfunctions of a vehicle exterior environment recognition device;

FIGS. 4A to 4C are diagrams illustrating traffic signs;

FIG. 5 is a flowchart illustrating a flow of vehicle exteriorenvironment recognition processing;

FIGS. 6A to 6D are diagrams illustrating color images acquired by animage acquiring module;

FIG. 7 is a diagram illustrating the Hough transform;

FIG. 8A to 8D are diagrams illustrating the Hough transform;

FIGS. 9A and 9B are diagrams illustrating a fourth extraction condition;

FIGS. 10A and 10B are flowcharts illustrating examples of feature pointidentifying processing;

FIGS. 11A and 11B are diagrams illustrating one example of the featurepoint identifying processing;

FIGS. 12A to 12C are diagrams illustrating voting processing;

FIGS. 13A to 13C are diagrams illustrating a vote table;

FIG. 14 is a diagram illustrating a center point candidate list;

FIG. 15 is a diagram illustrating a flag table;

FIG. 16 is a flowchart illustrating one example of traffic signidentifying processing;

FIGS. 17A and 17B are diagrams illustrating processing of a traffic signcorrecting module;

FIG. 18 is a flowchart illustrating a particular flow of traffic signcontent recognition processing;

FIG. 19 is a diagram illustrating a recognition target area;

FIG. 20 is a diagram illustrating a traffic sign that presents a removalof a speed limit;

FIG. 21 is a diagram illustrating vertical alignment processing;

FIG. 22 is a diagram illustrating templates;

FIG. 23 is a diagram illustrating horizontal matching processing;

FIG. 24 is a chart illustrating DP matching;

FIG. 25 is a diagram illustrating matching processing of an observingpart;

FIG. 26 is a diagram illustrating evaluation results;

FIG. 27 is a time chart illustrating a flow of a result notification ofa traffic sign;

FIGS. 28A to 28D are diagrams illustrating indication types of trafficsigns by country; and

FIG. 29 is a table illustrating templates of the traffic signs.

DETAILED DESCRIPTION

Hereinafter, suitable examples of the present disclosure will bedescribed in detail with reference to the accompanying drawings. Notethat dimensions, materials, particular numerical values, etc.illustrated in the examples are merely illustrations for easierunderstanding of the present disclosure and, thus, they are not to limitthe present disclosure unless otherwise particularly specified. Further,in this description and the drawings, elements having substantially thesame functions and configurations are denoted with the same referencenumerals for eliminating redundant explanations. Moreover, illustrationsof elements that are not directly related to the present disclosure areomitted herein.

Vehicle Exterior Environment Recognition System 100

FIG. 1 is a block diagram illustrating a connecting relation of avehicle exterior environment recognition system 100. The vehicleexterior environment recognition system 100 includes imaging devices110, a vehicle exterior environment recognition device 120, and avehicle control device 130 (e.g., an ECU or Engine Control Unit), whichare provided inside a vehicle 1. Note that, the vehicle 1 is simplyreferred to as “the vehicle” or “the vehicle 1” herein for the purposeof discriminating from other vehicles.

Each imaging device 110 is comprised of imaging elements, such as CCDs(Charge-Coupled Devices) and CMOSs (Complementary Metal-OxideSemiconductors). The imaging device 110 can image an environmentcorresponding to an area ahead of the vehicle 1 to generate a colorimage that can be expressed by color values. Here, the color value is anumerical group that is expressed by either of one of a YUV color spaceconsisting of one luminance (Y) and two color differences (U and V), anRGB color space consisting of three hues (R (Red), G (Green), and B(Blue)), or an HSB color space consisting of the hues (H), chroma (S),and brightness (B). In this example, a color image expressed by thecolor values of the YUV form will be described as an example image;however, a luminance image (monochrome image) that can be expressed bythe luminance (Y) can also be used in addition to the color image forapplications in which any partial processing can be carried out withoutdepending on the color image.

Moreover, the pair of imaging devices 110 is disposed so as to beseparated from each other in a substantially horizontal direction. Theimaging devices 110 are oriented in a traveling direction of the vehicle1, and optical axes thereof are substantially parallel to each other.Each imaging device 110 sequentially generates a color image that isobtained by imaging objects existing within a detection area ahead ofthe vehicle 1 frame by frame, for example, at the rate of 1/60 seconds(i.e., 60 fps). In this example, the objects to be recognized not onlyinclude solid objects that exist independently, such as vehicles,pedestrians, traffic signals, roads (traveling path), traffic signs,gates, guardrails, and buildings, but also include objects that can beidentified as part of the solid object, such as content of the trafficsign, a brake lamp, a high-mounted stop lamp, a taillight, a blinker,and each illuminating part of the traffic signal. Each functional modulein the following examples carries out respective processing for eachframe, triggered by a refresh of such a color image.

Further, in this example, each imaging device 110 images the detectionarea in a first exposure mode where an exposure time and an aperturestop according to brightness of the vehicle exterior environment (e.g.,measurements of an illuminometer) are defined and generates a firstimage. Each imaging device 110 also generates an image from which aparticular light source (e.g., a traffic sign of an electric lightdisplay type) self-emits light can be determined. The method thereforincludes using imaging elements having a large dynamic range and imagingso that black defects are not caused to objects that do not emit light,and halation is not caused to light sources, or imaging one detectionarea in a second exposure mode that is different in the exposure mode(i.e., the exposure time and the aperture stop) from the first exposuremode to generate a second image. For example, if it is daytime, thesecond image is generated with a shorter exposure time of the secondexposure mode than the exposure time of the first exposure mode that isdetermined according to the bright vehicle exterior environment, or witha smaller aperture. In this example, the first image and the secondimage are used as a color image and a distance image, respectively.Further, the first exposure mode and the second exposure mode areachieved as follows.

For example, the first image and the second image can be sequentiallygenerated by dividing the periodic imaging timing of the imaging device110, and alternately imaging in the first exposure mode and the secondexposure mode. Alternatively, two capacitors are provided for everypixel, and the imaging elements that can charge the two capacitors inparallel are provided. The time periods for charging the capacitors byone exposure are differentiated to parallelly generate two images thatare different in the exposure mode. Alternatively, the same purpose canbe achieved by reading twice at different timings during a charge of onecapacitor, and parallelly generating two images that are different inthe exposure mode. Alternatively, two sets of imaging devices 110 thatare different in the exposure mode may be prepared in advance (here, twosets of two imaging devices 110), and the two sets of the imagingdevices 110 may generate an image, respectively.

The vehicle exterior environment recognition device 120 acquires a colorimage from each of the two imaging devices 110. The vehicle exteriorenvironment recognition device 120 uses so-called pattern matching inwhich one block (e.g., a matrix of 4 pixels in horizontal directions×4pixels in vertical directions) extracted from one of the color images,and a block corresponding to the block in the first color image issearched in the second color image, to derive parallax information thatincludes parallax and a screen position indicative of the position ofthe block in question in the screen image. Note that the horizontaldirections indicate screen transverse or lateral directions of thecaptured image, and the vertical directions indicate screen verticaldirections of the captured image. The pattern matching includescomparing a pair of images in the luminance (Y) block by block. Forexample, the pattern matching includes SAD (Sum of Absolute Difference)in which a difference of the luminance is obtained, SSD (Sum of Squaredintensity Difference) in which the difference is squared before used,and NCC (Normalized Cross Correlation) in which an average vale ofluminances of pixels is calculated, and a variance is obtained bysubtracting the average value from the luminance of each pixel to findthe similarity. The vehicle exterior environment recognition device 120performs such parallax derivation processing block by block for all theblocks that are displayed within the detection area (e.g., 600pixels×200 pixels). Here, the block is comprised of 4 pixels×4 pixels;however, the number of pixels within one block can be suitably selected.

Note that, although the vehicle exterior environment recognition device120 can derive the parallax for every block that is a unit of detectionresolution, it cannot recognize what kind of object the block isinvolved. Therefore, the parallax information is independently derivednot object by object unit but by detection resolution by detectionresolution (e.g., block by block) in the detection area. Here, the imageassociated with the parallax information thus derived is referred to asthe distance image, in order to distinguish from the color imagedescribed above.

FIGS. 2A and 2B are diagram illustrating a color image 126 and adistance image 128, respectively. For example, suppose that the twoimaging devices 110 generate the color image 126 corresponding to thedetection area 124 as illustrated in FIG. 2A. Note that only one of thetwo color images 126 is schematically illustrated here in order tofacilitate understandings. The vehicle exterior environment recognitiondevice 120 obtains the parallax for every block from such a color image126, and forms the distance image 128 as illustrated in FIG. 2B. Eachblock in the distance image 128 is associated with the parallax of theblock. Here, the block of which the parallax is derived is expressed bya black dot for convenience of explanation. In this example, the colorimage 126 and the distance image 128 are generated based on the firstimage and the second image, respectively.

Further, the vehicle exterior environment recognition device 120 usesthe color values based on the color image 126, and three-dimensionalpositional information in the real space including a relative distancewith the vehicle 1 that is calculated based on the distance image 128 togroup the blocks that are equal in the color values and are close at thethree-dimensional positional information as an object. The vehicleexterior environment recognition device 120 then identifies to whichparticular object (e.g., a leading vehicle) the object in the detectionarea ahead of the vehicle 1 corresponds. For example, the vehicleexterior environment recognition device 120 can identify a leadingvehicle that travels forward based on the relative distance, etc., andcan further grasp the acceleration and deceleration of the leadingvehicle by correctly recognizing whether the stop lamps of the leadingvehicle are illuminated based on the color values. Moreover, the vehicleexterior environment recognition device 120 identifies a traffic signlocated at a road shoulder or a gate, further recognizes the content ofthe traffic sign (e.g., a speed limit), and then controls the speed ofthe vehicle 1 at a safe speed within the speed limit through the vehiclecontrol device 130.

Note that the relative distance can be found by converting the parallaxinformation for every block in the distance image 128 intothree-dimensional positional information by using a so-called stereomethod. Here, the stereo method is a method of deriving the relativedistance of the object with respect to the imaging devices 110 based onthe parallax of the object by using a triangulation method.

Returning to FIG. 1, the vehicle control device 130 receives operator'soperational inputs through a steering wheel 132, an accelerator or gaspedal 134, and a brake pedal 136, and transmits the operational inputsto a steering mechanism 142, a drive mechanism 144, and a brakemechanism 146, respectively, to control the vehicle 1. The vehiclecontrol device 130 also controls the drive mechanism 144 and the brakemechanism 146 according to instructions from the vehicle exteriorenvironment recognition device 120.

Next, a configuration of the vehicle exterior environment recognitiondevice 120 is described in detail. Here, particular processing of thetraffic sign that is a feature of this example is described in detail,and description of configurations unrelated to the feature of thisexample is omitted.

Vehicle Exterior Environment Recognition Device 120

FIG. 3 is a functional block diagram schematically illustratingfunctions of the vehicle exterior environment recognition device 120. Asillustrated in FIG. 3, the vehicle exterior environment recognitiondevice 120 is comprised of an I/F unit 150, a data holding unit 152, anda central controlling unit 154.

The I/F unit 150 is an interface that performs bidirectional informationexchanges with the imaging device 110 or the vehicle control device 130.The data holding unit 152 is comprised of one or more RAMs, one or moreflash memories, one or more HDDs, etc., and holds various informationrequired for the processing of each functional module described below.The data holding unit 152 temporarily holds images received from theimaging devices 110 (the color image 126 and distance image 128 based onthe first image and the second image).

The central controlling unit 154 is comprised of one or moresemiconductor integrated circuits that include one or more centralprocessing units (CPUs), one or more ROMs where one or more programs,etc. are stored, one or more RAMs as work areas, etc. The centralcontrolling unit 154 controls the I/F unit 150, the data holding unit152, etc. through a system bus 156. In this example, the centralcontrolling unit 154 also functions as an image acquiring module 160, apositional information deriving module 162, a feature point identifyingmodule 164, a voting module 166, a traffic sign identifying module 168,a traffic sign correcting module 170, a traffic sign content recognizingmodule 172, and a traffic sign content determining module 174. Next,traffic signs that are targets to be recognized in this example aredescribed, and, after that, vehicle exterior environment recognitionprocessing that is a feature of this example will be described indetail, considering operation of each functional module of the centralcontrolling unit 154.

FIGS. 4A to 4C are diagrams illustrating traffic signs. Some categoriesof the traffic signs are a traffic sign that presents a speed limit anda traffic sign that presents a removal of the speed limit. Asillustrated in FIG. 4A, the traffic sign that presents the speed limitis such that a numerical value indicative of the speed limit isindicated inside a circular frame line. The traffic signs that present aremoval of the speed limit are such that, as illustrated in FIG. 4B, aslanted line (either right side up or left side up) is indicated in aplain background, or as illustrated in FIG. 4C, the same slanted line isindicated over the background of the numerical value indicative of thespeed limit.

Indication types of the traffic sign are an electric light display typehaving one or more light sources, such as LEDs, and a non-electric lightdisplay type that is painted in different colors without having anylight sources. Further, installing locations of the traffic sign are ata road shoulder, and on a gate that is built in an arch shape betweenboth road shoulders (particularly, a location corresponding above theroad).

The vehicle exterior environment recognition device 120 of this examplerecognizes the contents of the traffic signs that differ in theinstalling locations, indication types, and categories, by thefunctional modules of the central controlling unit 154 described above.When the vehicle exterior environment recognition device 120 recognizesthe content of the traffic sign, it can inform a vehicle operator aboutthe content (for example, the speed limit of the currently-travelingroad, or the fact of overspeeding if the current speed is exceeding thespeed limit), or can control the vehicle control device 130 so as not toexceed the speed limit. Thus, there is not necessary to recognize thetraffic sign exactly when the vehicle 1 reaches the position where itcan examine the traffic sign, but it may be sufficient to do so when thevehicle just passed the traffic sign, or even after that. Therefore, itis sufficient to recognize the traffic sign over a plurality of framesand conclusively determine the content of the traffic sign based on theinformation of the plurality of frames.

Vehicle Exterior Environment Recognition Processing

FIG. 5 is a flowchart illustrating a flow of the vehicle exteriorenvironment recognition processing. The vehicle exterior environmentrecognition processing is roughly divided into and is performed in theorder as follows: image acquisition processing in which the images areacquired (S200); traffic sign detection processing in which a trafficsign (particularly, the circular frame line) is detected (S202); trafficsign content recognition processing in which the content of the trafficsign (the numerical value or a graphical pattern) is recognized (S204);and traffic sign content determining processing in which the content ofthe recognized traffic sign is conclusively determined by integratingwith time (S206).

Image Acquisition Processing S200

The image acquiring module 160 acquires the color images 126 from theimaging devices 110. As described above, there are different indicationtypes, such as the electric light display type and the non-electriclight display type, for the target traffic sign in this example, andthere are different installing locations, such as at the road shoulderand on the gate. Therefore, the imaging devices 110 image two detectionareas in two exposure modes (i.e., the first exposure mode and thesecond exposure mode) where the road shoulder and the gate can bedetected, respectively, and the image acquiring module 160 acquirestotal of four color images 126 thus acquired.

FIGS. 6A to 6D are diagrams illustrating the color images 126 acquiredby the image acquiring module 160. For example, the imaging devices 110image the color images 126 illustrated in FIGS. 6A and 6B in the firstexposure mode that is relatively long in the exposure time. Note thatFIG. 6A illustrates the color image 126 obtained by imaging at an angleof view that can detect the road shoulders, and FIG. 6B illustrates thecolor image 126 obtained by imaging after the angle of view is switchedwider than that of FIG. 6A in order to detect the gate. Then, theimaging devices 110 switch the exposure mode to the second exposure modein which, for example, the exposure time is made relatively shorter, andimage the color images 126 illustrated in FIGS. 6C and 6D. Note that thecolor images 126 of FIGS. 6C and 6D are imaged similarly to the colorimages 126 of FIGS. 6A and 6B before and after the angle of view isswitched according to the detection target. Here, although the fourcolor images 126 are illustrated, the number and type thereof can besuitably selected as long as the road shoulders and the gate aredetectable.

Thus, the four color images 126 of different exposure modes anddetection areas are acquired. That is, since the imaging is performed inthe plurality of exposure modes and for the plurality of detectionareas, troubles, such as luminances of light sources are saturated, andthe resolution becomes low due to an excessively-large angle of view,can be resolved and, thus, the detection accuracy can fully be improved.These four color images 126 are imaged in a time-division manner, andthe imaging order can suitably selected. Note that, in this example,since it is only necessary to recognize the traffic sign when or afterthe vehicle passes the traffic sign, it is not necessary to image thefour color images 126 at the same timing.

The positional information deriving module 162 acquires the color images(FIGS. 6A and 6B) based on the first images imaged at the first exposureby the two imaging devices 110, and derives the parallax informationincluding the parallaxes by using the pattern matching, and the screenpositions indicative of the positions of blocks in the screen, togenerate the distance image 128. Then, the positional informationderiving module 162 converts the parallax information for every blockwithin the detection area 124 in the distance image 128 by using thestereo method into three-dimensional positional information containing ahorizontal distance x that is a horizontal relative distance centeringon the horizontal center of the vehicle 1, a height y from a roadsurface, and a relative distance z with respect to the vehicle 1 in adepth direction. Note that the positional information deriving module162 obtains a vertical position of the road surface in advance beforeconverting into the three-dimensional positional information, andderives the height y from the road surface based on the relativedistance between the vertical position of each block and the verticalposition of the road surface. Here, the parallax information indicatesthe parallax of each block in the distance image 128, while thethree-dimensional positional information indicates the information onthe relative distance z of each block in real space. If the parallaxinformation is derived not pixel by pixel but by block by block (i.e.,by a plurality of pixel units), the calculation can be performed pixelby pixel, considering that the parallax information applies to all thepixels belonging to the block concerned. For the conversion into thethree-dimensional positional information, since known arts such asJapanese Unexamined Patent Application Publication (JP-A) No.2013-109391 can be referred, detailed description thereof is omittedherein.

Traffic Sign Detection Processing S202

Particularly in this example, the target of recognition is a circulartraffic sign among others. Such a circular traffic sign is detected byusing the Hough transform. Here, the Hough transform is a technique tovote for the feature points on the color image where an object possiblyexists to detect the object with the large number of votes (equal to ormore than a predetermined value). Thus, although the Hough transform isparticularly described in this example, various known shape recognitionapproaches, such as template matching and least-squares method, can alsobe used other than the Hough transform, for applications in which thetraffic sign can be identified in any partial processing of the vehicleexterior environment recognition processing without depending on theHough transform.

FIG. 7 and FIGS. 8A to 8D are diagrams illustrating the Hough transform.Here, suppose that three pixels 220 c, 220 d, and 220 e having an edgeare extracted from the color image 126 as illustrated in the part (a) ofFIG. 7. Originally, although these three pixels 220 c, 220 d, and 220 eare part of a circular traffic sign 222, that the traffic sign has acircular shape cannot normally be clearly grasped from the color image126.

The Hough transform is an approach of detecting a geometric shape, suchas a circle and a straight line, from a plurality of points, and it isbased on theory that the center of a circle that passes through anarbitrary pixel 220 and has a radius n exists on the circumference ofthe radius n centering on the arbitrary pixel 220. For example, thecenter of the circle that passes through the three pixels 220 c, 220 d,and 220 e in the part (a) of FIG. 7 is on the circumferences centeringon the three pixels 220 c, 220 d, and 220 e. However, since the radius ncannot be identified based on the information only on the edge, aplurality of radii n that are different from each other are prepared,pixels on the circles of the plurality of radii n centering on the threepixels 220 c, 220 d, and 220 e are voted, and if the number of votesobtained becomes equal to or greater than the predetermined value, theradius n and the center are determined to be the traffic sign 222.

For example, as illustrated in the parts (b), (c), and (d) of FIG. 7,circles having different radii n=4, 5, and 6 are formed centering on thethree pixels 220 c, 220 d, and 220 e, and the pixels contained in theloci of the circles are voted (unit indices are associated). Then, inthe part (b) of FIG. 7, the number of votes obtained becomes 2 at twopixels 224 (two unit indices are associated). Further, in the part (c)of FIG. 7, the number of votes obtained becomes 2 at three pixels 224,and the number of votes obtained becomes 3 at one pixel 226. Similarly,in the part (d) of FIG. 7, the number of votes obtained becomes 2 at sixpixels 224.

At this time, it is only the pixel 226 of which the number of votesobtained becomes 3 (i.e., equal to or greater than the predeterminedvalue), the pixel 226 is used as the center of a circle that passesthrough the three pixels 220 c, 220 d, and 220 e, and the radius n=5 atthe time of deriving the pixel 226 concerned can be identified as theradius of the circle. Thus, as illustrated in the part (e) of FIG. 7, acircle 228 that passes through the three pixels 220 c, 220 d, and 220 eis identified. Here, although the three pixels 220 c, 220 d, and 220 eare described as one example for convenience of explanation, since apixel that is not contained in the circle 228 may be used as the featurepoint or the pixel that appears at a position different from an originalposition due to pixelization (dispersion) may be used as the featurepoint, a number of points are actually used for the votes in order toavoid the effects of such noise and, thus, a stable detection can beperformed by majority rule. In this example, such Hough transform isapplied to the color image 126 illustrated, for example, in FIG. 6B, andthe circular traffic sign is identified by the feature point identifyingmodule 164, the voting module 166, and the traffic sign identifyingmodule 168. Next, fundamental processing of each functional module isdescribed with reference to FIGS. 8A to 8D.

First, the feature point identifying module 164 identifies the featurepoint corresponding to part of the circumference based on the colorimage 126 (feature point identifying processing). For example, supposethat the feature point identifying module 164 identifies pixels 220 f,220 g, 220 h, 220 i, 220 j, and 220 k having edges as feature points inthe color image 126 of FIG. 8A. Note that the dotted line in FIG. 8Acorresponds to a traveling lane.

Next, the voting module 166 votes for a predetermined distancecorresponding to the radius n from the feature points (votingprocessing). Here, for the six pixels 220, the radius n is temporarilyset to 30 pixels for the pixels 220 f, 220 g, and 220 h, and the radiusn is temporarily set to 23 pixels for the pixels 220 i, 220 j, and 220k, for convenience of explanation. The voting module 166 votes, in thecolor image 126 of FIG. 8B, for all the pixels on circles having theradius n (30 pixels and 23 pixels) from the pixels 220, centering on thepixels 220 f, 220 g, 220 h, 220 i, 220 j, and 220 k, respectively. Then,the voting targets (pixel and the radius n) in the vote table 230illustrated in FIG. 8C are voted (added) by 1. The vote table 230 is avoting space by the Hough transform, and it is expressed by threedimensions of the screen positions (x, y) of the pixel and the radii nthat are used as the voting targets.

Next, the traffic sign identifying module 168 detects the number ofvotes obtained in the vote table 230, and derives the center and theradius n of the circle based on the pixel and the radius n of the votingtarget with a large number of votes obtained. Then, as illustrated inFIG. 8D, a circle 236 of the radius n that is read centering on thepixel 234 with a large number of votes obtained is formed, and thecircle is identified as a traffic sign (traffic sign identifyingprocessing). Next, particular operations of the feature pointidentifying processing, the voting processing, and the traffic signidentifying processing performed by the feature point identifying module164, the voting module 166, and the traffic sign identifying module 168,respectively, are described.

Feature Point Identifying Processing

The feature point identifying module 164 uses the color image 126, andselects one pixel 220 having predetermined edge intensity among thepixels 220 as a candidate of the feature point, which serves as a firstextraction condition. The edge intensity may be expressed, for example,by the Sobel filter. Assume that the coordinates of each pixel 220 are(i, j) and the luminance is A(i, j), the feature point identifyingmodule 164 uses the following Equation 1 to derive the sum of absolutevalues of a vertical Sobel filter and a horizontal Sobel filter, andselects the pixel (i, j) as a candidate of the feature point if the sumvalue (edge intensity) is equal to or greater than the predeterminedvalue.Edgeintensity=|A(i+1,j+1)+2A(i+1,j)+A(i+1,j−1)−A(i−1,j+1)−2A(i−1,j)−A(i−1,j−1)|+|A(i+1,j+1)+2A(i,j+1)+A(i−1,j+1)−A(i+1,j−1)−2A(i,j−1)−A(i−1,j−1)|  (Equation 1)

Here, although one example in which the edge intensity is derived by theSobel filter is described, various known techniques, such as the Prewittfilter, can also be applied without limiting to the Sobel filter.

Further, the feature point identifying module 164 uses the color image126, and selects one pixel 220 among the pixels 220 as a candidate ofthe feature point, which serves as a second extraction condition, if apredetermined color component of the predetermined color values of theselected pixel 220 (e.g., a V-component in the color space of YUV form)is equal to or greater than the predetermined value. The traffic signthat presents a speed limit is comprised of a red circle along thecircumference, and the traffic sign that presents a removal of the speedlimit is comprised of a white circle or a black circle along thecircumference. Therefore, only the color belonging to an area where theV-component is equal to or greater than the predetermined value isextracted, and it is used as a candidate of the feature point. Thus, thegreen pixels, such as trees, which can be observed often duringtraveling, can be excluded, and a suitable narrowing of the featurepoints becomes possible.

Note that, if the color image is comprised of a color space of RGB form,it is converted into a color space of YUV form by a suitable conversion.Since such conversions are known arts, detailed description is omittedherein.

The feature point identifying module 164 uses the distance image 128,and selects one pixel 220, which satisfies any one or more conditions,among the pixels 220 as a candidate of the feature point, which servesas a third extraction condition, where the conditions are the relativedistance z is within a predetermined range, the height y from a roadsurface is within a predetermined range, and the horizontal distance xis within a predetermined range.

Particularly, the feature point identifying module 164 extracts onepixel 220 from the distance image 128, refers to the three-dimensionalpositional information on the pixel 220, and if the relative distance zof an object corresponding to the pixel 220 is located, for example,equal to or higher than 10 m and less than 50 m, the feature pointidentifying module 164 selects the pixel 220 as the candidate of thefeature point. This is because a traveling distance of the object on theimage within the exposure time becomes longer if the object is locatedlower than 10 m, and the effects of a blur of the image becomes greateraccordingly. Further, if the object is located equal to or higher than50 m, the content of the traffic sign cannot often be correctlyrecognized because of the resolution of the image. Thus, the processingload and erroneous recognition can be reduced by limiting the relativedistance z.

Further, if the height y of the object from the road surfacecorresponding to the pixel 220 concerned is located, for example, equalto or higher than 0.5 m and lower than 6.0 m, the feature pointidentifying module 164 selects the pixel 220 as a candidate of thefeature point. Because the range is set to “equal to or higher than 0.5m,” road markings and lanes can be excluded from the recognition target,and because the range is set to “lower than 6.0 m,” trees or the likelocated higher can be excluded. The processing load and the erroneousrecognition can be reduced by the conditions concerned.

Further, if the horizontal distance x of the object corresponding to thepixel 220 concerned is located, for example, within a range of 12 m(equal to or higher than −12 m and lower than 12 m), the feature pointidentifying module 164 selects the pixel 220 as the candidate of thefeature point. Traffic signs other than the traffic signs related to thelane where the vehicle 1 is traveling can be excluded by setting therange to 12 m. The processing load and the erroneous recognition can bereduced by the conditions concerned.

Moreover, the feature point identifying module 164 uses the color image126 and the distance image 128, and selects pixels 220 as candidates ofthe feature point, which serve as a fourth extraction condition, wherethe pixels 220 are adjacent pixels of which a difference of at least onecolor component (e.g., U-component) is within a predetermined range, andsuch pixels 220 are not successively located equal to or greater than apredetermined distance (length) in a predetermined direction.

FIGS. 9A and 9B are diagrams illustrating the fourth extractioncondition. For example, a tree 240 with autumnal red leaves (turned-redtree) or the like that is illustrated at a road shoulder in FIG. 9A maysatisfy all of the first to third extraction conditions. Since theturned-red tree 240 has many parts that can be recognized as textures,it may often be extracted as feature points over a large area thereof.Therefore, feature points are identified over the large area of the tree240 or the like with many textures, and the processing load increases.

For this reason, the feature point identifying module 164 determineswhether the distance between the pixels 220, that is, any one of thedepth distance, the vertical distance, and the horizontal distance, or asynthetic distance of any two or more is, for example, shorter than thepredetermined distance (e.g., 0.5 m), and the difference of one colorcomponent (e.g., U-component) is equal to or less than the predeterminedvalue (e.g., 10). Then, if the pixels 220 of which the syntheticdistance is shorter than 0.5 m and the difference of U-component isequal to or less than 10 continue for 30 pixels in one direction (e.g.,horizontal direction) of the screen, the feature point identifyingmodule 164 excludes all the pixels 220 from the feature points. Asillustrated in FIG. 9B, the circumference of the traffic sign iscomprised of a circle of the same color, but the pixels do not continuefor 30 pixels the predetermined direction, for example, the horizontaldirection or the vertical direction. Since the feature points areidentified based on such characteristics of the indication type of thetraffic sign, the pixels 220 that should not originally be extracted asthe feature points but that satisfy the first to third extractionconditions can be excluded, and the identifying efficiency of thefeature points can be improved.

Note that the size of the traffic sign varies in the color image 126depending on the relative distance z. Thus, the feature pointidentifying module 164 may change the predetermined distance that is athreshold for determining whether the pixels 220 continue in thepredetermined direction, according to the relative distance z withrespect to the vehicle 1. Particularly, the predetermined distance ismade longer as the relative distance z becomes shorter, and, on theother hand, the predetermined distance is made shorter as the relativedistance z becomes longer. By doing so, a suitable threshold can beprovided according to the size of the traffic sign in the color image126, and it becomes possible to appropriately exclude the pixels 220that should not originally be extracted as the feature points but thatsatisfy the first to third extraction conditions.

Then, the feature point identifying module 164 identifies the pixels 220that satisfy the first or second extraction condition as the featurepoints among the pixels 220 that satisfy both the third and fourthextraction conditions. Thus, the pixels 220 suitable for the featurepoints are identified.

The feature point identifying module 164 may suspend the feature pointidentifying processing concerned in one frame when the number of featurepoints becomes equal to or more than a predetermined value. The colorimage 126 may change variously according to the vehicle exteriorenvironment, and the number of feature points may increase dramaticallyaccording to the imaged environment. If the number of feature pointsthus increases, the processing load increases accordingly, and theprocessing time may exceed a time period assigned to one frame.Therefore, when the number of feature points becomes equal to or greaterthan the predetermined value, the feature point identifying module 164suspends the feature point identifying processing concerned in oneframe, and carries out the voting processing and subsequent processingonly for the feature points identified by this time point.

Note that, since the traffic sign is often located comparativelyupwardly in the color image 126, the feature point identifying module164 identifies the feature points sequentially from the upper part ofthe color image 126. Thus, it becomes possible to appropriately extractthe feature points corresponding to the part of the circumference of thetraffic sign.

In order to keep the number of feature points equal to or less than thepredetermined value as described above, the feature point identifyingmodule 164 may change the predetermined value of the edge intensity onthe first extraction condition and the predetermined value ofV-component on the second extraction condition for each frame. Since thevehicle exterior environment does not change much between frames, thenumber of feature points does not change so much, either. Therefore,when many feature points are extracted in one frame, many feature pointsare extracted continuously also in subsequent frames. For this reason,the predetermined value of the edge intensity on the first extractioncondition and the predetermined value of V-component on the secondextraction condition are adjusted within a predetermined range (40 to150), while the number of feature points is kept within a predeterminedrange (200 to 2000) so that the processing time does not exceed the timeperiod assigned to the identification of the feature points in oneframe.

FIGS. 10A and 10B are flowcharts illustrating examples of the featurepoint identifying processing. As illustrated in FIG. 10A, the featurepoint identifying module 164 determines whether the number of featurepoints extracted in one frame on the first extraction condition exceedsthe feature point upper limit (here, 2000) (S240). As a result, if thenumber of feature points exceeds the feature point upper limit (YES atS240), the feature point identifying module 164 determines whether thepredetermined value of the edge intensity is less than the edgeintensity upper limit (here, 150) (S242). As a result, if thepredetermined value of the edge intensity is less than the edgeintensity upper limit (YES at S242), the feature point identifyingmodule 164 increments the predetermined value of the edge intensity(S244). The predetermined value of the edge intensity is reflected tothe feature point identifying processing of the subsequent frames. Ifthe predetermined value of the edge intensity is equal to or greaterthan the edge intensity upper limit (NO at S242), the feature pointidentifying module 164 does not increment the predetermined value of theedge intensity, but maintains the predetermined value that reached theedge intensity upper limit (it is maintained within the predeterminedrange).

At Step S240, if the number of feature points is equal to or less thanthe feature point upper limit (NO at S240), the feature pointidentifying module 164 determines whether the number of feature pointsextracted in one frame is less than the feature point lower limit (here200) (S246). As a result, if the number of feature points is less thanthe feature point lower limit (YES at S246), the feature pointidentifying module 164 determines whether the predetermined value of theedge intensity exceeds the edge intensity lower limit (here, 40) (S248).As a result, if the predetermined value of the edge intensity exceedsthe edge intensity lower limit (YES at S248), the feature pointidentifying module 164 decrements the predetermined value of the edgeintensity (S250). If the predetermined value of the edge intensity isequal to or less than then edge intensity lower limit (NO at S248), thefeature point identifying module 164 does not decrement thepredetermined value of the edge intensity, but maintains thepredetermined value that reached the feature point lower limit (it ismaintained within the predetermined range). If the number of featurepoints is equal to or greater than the feature point lower limit (NO atS246), the feature point identifying module 164 does not carry out anyprocess.

As illustrated in FIG. 10B, the feature point identifying module 164determines whether the number of feature points extracted in one frameon the second extraction condition exceeds the feature point upper limit(here, 2000) (S260). As a result, if the number of feature pointsexceeds the feature point upper limit (YES at S260), the feature pointidentifying module 164 determines whether the predetermined value ofV-component is less than the V-component upper limit (here, 150) (S262).As a result, if the predetermined value of V-component is less than theV-component upper limit (YES at S262), the feature point identifyingmodule 164 increments the predetermined value of V-component (S264). Thepredetermined value of V-component is reflected to the feature pointidentifying processing of the subsequent frames. If the predeterminedvalue of V-component is equal to or greater than the V-component upperlimit (NO at S262), the feature point identifying module 164 does notincrement the predetermined value of V-component, but maintains thepredetermined value that reached the feature point upper limit (it ismaintained within the predetermined range).

At Step S260, if the number of feature points is equal to or less thanthe feature point upper limit (NO at S260), the feature pointidentifying module 164 determines whether the number of feature pointsextracted in one frame is less than the feature point lower limit (here,200) (S266). As a result, if the number of feature points is less thanthe feature point lower limit (YES at S266), the feature pointidentifying module 164 determines whether the predetermined value ofV-component exceeds the V-component lower limit (here, 40) (S268). As aresult, if the predetermined value of V-component exceeds theV-component lower limit (YES at S268), the feature point identifyingmodule 164 decrements the predetermined value of V-component (S270). Ifthe predetermined value of V-component is equal to or less than theV-component lower limit (NO at S268), the feature point identifyingmodule 164 does not decrement the predetermined value of V-component,but maintains the predetermined value that reached the feature pointlower limit (it is maintained within the predetermined range). If thenumber of feature points is equal to or greater than the feature pointlower limit (NO at S266), the feature point identifying module 164 doesnot carry our any process.

As described above, in this example, the pixels 220 that satisfy eitherone of the first extraction condition and the second extractioncondition are identified as the feature points. Therefore, the number offeature points is independently adjusted for the first extractioncondition and the second extraction condition as illustrated in FIGS.10A and 10B. Thus, the processing time assigned to the identification ofthe feature points in one frame is maintained by keeping thepredetermined value of the edge intensity on the first extractioncondition or the predetermined value of V-component on the secondextraction condition within the predetermined range (40 to 150), whileadjusting the number of feature points within the predetermined range(200 to 2000).

The feature point identifying module 164 may change the predeterminedvalue of V-component on the second extraction condition for each frameaccording to the color component of a road surface. The color componentthroughout the color image 126 varies according to sunlight condition orlighting environment. For example, in a tunnel where orange lightingsare installed, the V-component throughout the color image 126 increases.Therefore, the feature point identifying module 164 changes thepredetermined value of V-component on the second extraction conditionaccording to the color component of the road surface to reduce theeffects of the changes of sunlight or lighting against theidentification of the feature points.

FIGS. 11A and 11B are diagrams illustrating one example of the featurepoint identifying processing. As illustrated in FIG. 11A, the featurepoint identifying module 164 acquires the color values of RGB form at 4points 280 of the predetermined relative distance z where a road surfaceahead of the vehicle is highly-possibly displayed in the color image126. Then, the feature point identifying module 164 obtainsG-component/R-component at each of the 4 points 280, and derives anaverage value AV of the values. Next, the feature point identifyingmodule 164 multiplies the R-components of all the pixels in the colorimage 126 by the average value AV, converts them into YUV form, comparesthe V-component after the conversion with a predetermined value, andidentifies the candidates of the feature point.

Note that, as illustrated in FIG. 11B, if the relative distance z of aleading vehicle recognized by the vehicle exterior environmentrecognition device 120 is within a predetermined range (e.g., equal toor less than 20 m), the feature point identifying module 164 does notcalculate the average value AV from the color image 126, but usesanother average value AV that is derived (used) for the previous frame.This is for avoiding that the color component of the leading vehicle isacquired as the color component of the road surface, this affects to theV-component, and the candidates of the feature point are erroneouslyextracted.

Moreover, regardless of the average value AV calculated for the currentframe is not changed in the exposure mode from the average value AVderived (used) for the previous frame (there is no significant change insurrounding brightness), the average value AV is not calculated from thecolor image 126, but the average value AV derived (used) for theprevious frame is used also when the change is equal to or greater thana predetermined value (e.g., ±50%). This is for avoiding that, when theroad surface is painted in red, this affects to the V-component, and thecandidates of the feature point are erroneously extracted. Note that,when shadows cover the road surface, since the color of gray influencesequally on each color component of RGB form (R, G, and B) and does notaffect to the value of G-component/R-component, this does not become aproblem.

Voting Processing

The voting module 166 votes for the circumferences that are apart by theradius n from the feature points identified by the feature pointidentifying module 164. This is based on that, assuming the featurepoints correspond to parts of the circumferences, the centers of thecircles of which the parts of the circumferences are the feature pointsmust be located on the circumferences of the radius n from the featurepoints. Therefore, the voting module 166 further votes for the points ofthe radius n from the feature points, which may be the centers of thecircles having the feature points as the parts of the circumferences.

FIG. 12 is a diagram illustrating the voting processing. As illustratedin FIG. 12A, if the radius n is described as one example, the votingmodule 166 normally uses all the pixels 220 on the circumference 302 ofthe radius n centering on a feature point 300 as corresponding points304, and votes for points of the radius n at the screen position.However, since such corresponding points 304 of the radius n are all thepixels 220 on the circumference 302, they become too many, and if theradius n is varied, the number will be substantially infinitely toomany. Therefore, the number of corresponding points 304 and the numberof radii n corresponding to one radius n are limited in order to improvethe efficiency of vote.

Here, it is known that the tangent of a circle is perpendicular to aline segment that connects between the center of the circle and thepoint of tangency. Further, the tangent of the circle corresponds to theedge extending direction of the pixels 220. Thus, the correspondingpoints 304 only appear on a line segment that is perpendicular to theedge extending direction of the feature point 300. Therefore, the votingmodule 166 can grasp the edge extending direction of the feature point300, and define the corresponding points 304 in a directionperpendicular to the edge extending direction.

Here, if the luminance A at the coordinates (i, j) of each pixel 220 isindicated as A(i, j), the voting module 166 derives a line segmentperpendicular to the edge extending direction based on a ratio ofabsolute values of a vertical Sobel filter and a horizontal Sobelfilter, as illustrated in the following Equation 2.Line segment perpendicular to edge extendingdirection=atan(|A(i+1,j+1)+2A(i,j+1)+A(i−1,j+1)−A(i+1,j−1)−2A(i,j−1)−A(i−1,j−1)|/|A(i+1,j+1)+2A(i+1,j)+A(i+1,j−1)−A(i−1,j+1)−2A(i−1,j)−A(i−1,j−1)|)  (Equation 2)

Here, although the example in which the line segment perpendicular tothe edge extending direction is derived by using the Sobel filter isdescribed, various known techniques can also be applied without limitingto the Sobel filter. Further, although the division and the arc tangent(atan) are used in Equation 2, if the processing load increases bythese, a look-up table from which a unique line segment perpendicular tothe edge extending direction can be derived using inputs of the absolutevalues of the vertical Sobel filter and the horizontal Sobel filter maybe used.

For example, as illustrated in FIG. 12B, if the edge extending directionof the feature point 300 is illustrated by a line segment 306 of adashed line, corresponding points 304 can be defined on a line segment308 of a dashed dotted line in a direction perpendicular to the linesegment 306. Here, if one radius n is described, the correspondingpoints 304 can be narrowed down to two points that are apart from thefeature point 300 by the radius n, and which are located on the linesegment 308 in a direction perpendicular to the edge extending directionof the feature point 300.

The traffic signs may be defined in one or more sizes by laws and/orrules of each country. Thus, the size of the traffic sign in the colorimage 126 can be defined based on the relative distance z. Therefore,the voting module 166 estimates the size of the traffic sign (radius n)in the color image 126 by using a inverse function of the function usedfor deriving the three-dimensional positional information, according tothe relative distance z, and narrows the number of radii n to be usedfor vote. For example, if the traffic sign that presents the speed limitor the traffic sign that presents a removal of the speed limit islimited to three sizes, the corresponding points 304 are narrowed downto that number (3)×2, as illustrated in FIG. 12C.

Thus, since the corresponding points 304 of one radius n are limited onthe line segment 308 in a direction perpendicular to the edge extendingdirection of the feature point 300, and the number of radii n is limitedto one or more according to the predetermined sizes and the relativedistance z, unwilling votes at which the corresponding points 304 shouldnot originally exist can be avoided. Therefore, the erroneous detectionsof the traffic sign due to the erroneous setting of the correspondingpoints 304 can be prevented, while avoiding the needless Hough transformprocessing and reducing the processing load.

The voting module 166 votes in the vote table 230 after limiting thecorresponding points 304 as described above. Although thethree-dimensional voting space is described herein, a voting space of Mdimension (M is a positive integer) that is extended in a lateral orvertical dimension (e.g., rotation) can also be formed in order to applyit to laterally-oriented traffic signs or inclined traffic signs.

FIGS. 13A to 13C are diagrams illustrating the vote table 230. The votetable 230 is normally comprised of the three-dimensional voting space([horizontal number of pixels H]×[vertical number of pixels V]×[value Nthat the radius n can take] in the color image 126) as illustrated inFIG. 13A, and the number of votes obtained is held at athree-dimensional position (point) of the pixel that serves as thevoting target and that is comprised of the screen position (x, y) withthe radius n. For example, if the maximum number of votes obtained is255 (1 byte), the size of the vote table 230 is H×V×N (bytes). In such acase, if the color image 126 of high resolution is used, a problem thatthe storage area of the memory required for the vote table 230 becomeslarge is caused, and, if the number of votes obtained is limited (i.e.,few), a problem that a peak of the number of votes obtained is difficultto appear due to the effects of noise, etc. is caused.

For the latter problem, it can be resolved by performing votingprocessing with added margins considering the noise, for example, votingfor radii n near the corresponding point 304 in addition to the radius nof the corresponding points 304, a new problem in which the processingload increases accordingly will be caused. Alternatively, the resolutionmay be lowered, and, for example, the voting may be performed by blockby block of 2 pixels in the horizontal directions×2 pixels in thevertical directions, a degradation of the identification accuracy of thetraffic sign at the corresponding point 304 will not be avoidablebecause of the lowered resolution.

Therefore, in this example, two vote tables (a first vote table 230 aand a second vote table 230 b) having different dimensions andresolutions are provided, and the voting module 166 votes in the votetables 230 a and 230 b simultaneously.

As illustrated in FIG. 13B, the vote table 230 a is represented by thetwo-dimensional voting space (horizontal pixel position and verticalpixel position) that is reduced in the dimension by one, where thehorizontal number of pixels H×the vertical number of pixels V remain forthe resolution of the color image 126 and from which the information onthe radius n is omitted. Therefore, the size of the vote table 230 a isH×V (bytes), where the number of votes obtained for all the radii n isheld at the two-dimensional positions of the screen positions (x, y) ofthe pixels that serve as the voting targets. On the other hand, asillustrated in FIG. 13C, the vote table 230 b uses the same resolution(i.e., the value N) for the radius n, and does not omit any dimension.Instead, the vote table 230 b compresses the horizontal and verticalresolutions of the color image 126 into one fourth (reducing theresolutions), and is illustrated by a three-dimensional voting space inwhich [horizontal number of pixels H/4 (the compressed value of thehorizontal number of pixels)]×[vertical number of pixels V/4 (thecompressed value of the vertical number of pixels)]. Therefore, the sizeof the vote table 230 b is H/4×V/4×N (bytes), and the number of votesobtained is held at the three-dimensional positions comprised of theblocks (4 pixels in the horizontal directions×4 pixels in the verticaldirections) to which the screen positions (x, y) of the pixels belongand the radius n, which serve as the voting target. Thus, the votetables 230 a and 230 b intentionally lowers the resolution of the radiusn, and intentionally lowers the resolution of the screen position,respectively.

The voting module 166 simultaneously votes for the vote tables 230 a and230 b when it derives the corresponding points 304 based on the featurepoint 300. However, as for the vote table 230 a, the voting module 166votes for one point corresponding to the corresponding point 304regardless of the radius n, and as for the vote table 230 b, the votingmodule 166 votes for the points of the radius n of the blocks to whichthe corresponding points 304 belong. Thus, when the votes are finished,the voting module 166 can select a point with a large number of totalvotes obtained (corresponding point 304) of the radius n in the votetable 230 a as the candidate of the center position of the traffic sign,and can select a radius n with a large number of votes obtained withinthe block corresponding to the center position concerned in the votetable 230 b as the candidate of the radius n of the traffic sign.

Thus, the total storage capacity of the vote table 230 can be reduced toH×V+H/4×V/4×N (bytes), while maintaining the identification accuracy ofthe center of the traffic sign with high accuracy. Here, if H=600pixels, V=200 pixels, and N=20 pixels, 600×200×20=2,400,000 bytes areoriginally required, it can be reduced to 600×200+600/4×200/4×20=270,000bytes equivalent to about 1/10 of the original capacity.

After the voting module 166 finishes the voting processing at all thefeature points, it extracts the number of votes obtained from each pointin the vote table 230, and selects the corresponding points 304 at whichthe total number of votes obtained of the radius n becomes equal to orgreater than the predetermined value as the candidates of the centerpoint of the traffic sign. However, even though the storage capacity ofthe vote table 230 is reduced, the determination of whether the numberof votes obtained in the entire vote table 230 is large or small stilltakes a large processing load when the voting space is large to someextent. Therefore, the voting module 166 selects the candidate of thecenter point in parallel to the voting to improve the extractionefficiency of the candidate of the center point.

FIG. 14 is a diagram illustrating a center point candidate list 310.Here, the center point candidate list 310 is provided other than thevote table 230. At least the screen positions (x, y) for the vote table230 a are registered with the center point candidate list 310. Wheneverthe number of votes obtained in the vote table 230 a at thecurrently-voting point reaches or becomes greater than a predeterminednumber, the voting module 166 additionally registers the correspondingpoint 304 corresponding to that point with the center point candidatelist 310. Then, after the voting processing is finished at all thefeature points, the voting module 166 selects only the correspondingpoints 304 that are registered with the center point candidate list 310as the candidates of the traffic sign.

With this configuration, it is possible to appropriately extract thecandidates of the center point, while avoiding the determination ofwhether the number of votes obtained in the entire vote table 230 islarge or small (i.e., while reducing the processing load). Note that thevoting module 166 limits the candidates of the center point that areregistered with the center point candidate list 310 by up to thepredetermined value (e.g., 50). This is based on the following reasons.That is, if the corresponding points 304 are distributed over thecandidates of a plurality of center points due to the effects of noise,etc., it is originally a single traffic sign but a plurality of centerpoints may be selected as the candidates. In such a case, an infinitenumber of center points should not be extracted as the candidatesbecause the possibility that 50 or more traffic signs exist in the colorimage 126 is normally small. Thus, when the center point candidates inthe center point candidate list 310 reach or become greater than 50, thevoting module 166 suspends the voting processing concerned in one frame,and applies the traffic sign identifying processing and subsequentprocessing only to the center point candidates that have been identifiedby this time point.

Thus, the center point candidate list 310 is generated by the votingmodule 166, and the center point candidate list 310 is also associatedwith the information of the radius, the number of votes obtained in thevote tables 230 a and 230 b, and the three-dimensional position of thepixel concerned, other than the screen position (center position).

When the votes are made with the vote tables 230 a and 230 b in such amanner, the voting module 166 initializes each point of the vote tables230 a and 230 b for the vote of the next frame so that the numbers ofvotes obtained are set to 0. However, the processing time required forthe initialization of the vote tables 230 a and 230 b cannot bedisregarded depending on the resolution of the color image 126, and itmay occupy 40% of the entire sign detection processing S202. Since morestorage capacity of the memory is taken up as the number of dimensionsincreases in the vote tables 230 a and 230 b, the effects of the load ofthe initialization processing become particularly large for thethree-dimensional vote table 230 b.

FIG. 15 is a diagram illustrating a flag table 320. Here, the flag table320 is provided in addition to the vote tables 230 a and 230 b. Asillustrated in the part (a) of FIG. 15, the flag table 320 is a table inwhich the number of dimensions is reduced by one dimension of the radiusn, and a flag is set at the two-dimensional position of the block (4pixels in the horizontal directions×4 pixels in the vertical directions)to which the screen position (x, y) of the pixel that serves as thevoting target belongs. Therefore, the size of the flag table 320 isH/4×V/4 (bytes).

When the voting module 166 votes for any one of radii n of any one ofblocks in the vote table 230 b as illustrated by cross-hatching in thepart (b) of FIG. 15, it changes the flag to ON of a block equivalent tothe block in the flag table 320 illustrated by cross-hatching in thepart (a) of FIG. 15. Therefore, when any one of the radii n in eachblock of the vote table 230 b is voted, the block of the flag table 320corresponding to the block of the vote table 230 b is changed to ON.Then, when the votes to the vote tables 230 a and 230 b are finished,the voting module 166 initializes each point of only N blocks in thevote table 230 b illustrated by hatching in the part (b) of FIG. 15,which correspond to the blocks at which the flags are ON in the flagtable 320 illustrated by hatching in the part (a) of FIG. 15, so thatthe number of votes obtained is set to 0. In other words, the votingmodule 166 does not perform the initialization processing for otherblocks in the vote table 230 b corresponding to the block at which theflag is OFF in the flag table 320.

Here, although the vote table 230 b is described as the target of theflag table 320, the concept of the flag table 320 can also be applied tothe vote table 230 of which the size is H×V×N (bytes), without anylimitation. In such a case, the size of the flag table 320 is H×V(bytes).

Alternatively, instead of providing the flag table 320, it may bedetermined whether the votes are performed for the area corresponding toeach block in the vote table 230 a, if the votes are performed, eachpoint of only the blocks in the vote table 230 b corresponding to thatblocks may be initialized so that the number of votes obtained is set to0. Thus, since it is unnecessary to initialize all points in the votetable 230 b, the load of the initialization processing can besignificantly reduced.

Alternatively, when the votes for the vote tables 230 a and 230 b arefinished, the voting module 166 may initialize each point of only aplurality of pixels (4 pixels in the horizontal directions×4 pixels inthe vertical directions) in the vote table 230 a corresponding to theblocks at which the flags are ON in the flag table 320 as illustrated byhatching in the part (c) of FIG. 15, so that the number of votesobtained is set to 0. Thus, since it is unnecessary to initialize allpoints in the vote table 230 a, similar to the vote table 230 b, theload of the initialization processing can be significantly reduced.

Then, after the initialization processing is finished, the voting module166 initializes all the flags in the flag table 320 to OFF. Thus, thevote table can be appropriately initialized without increasing the loadof the initialization processing.

Traffic Sign Identifying Processing

The traffic sign identifying module 168 narrows down the candidates ofthe traffic sign derived by the voting module 166 based on the first tothird narrowing conditions, and identifies the traffic sign.

The traffic sign identifying module 168 narrows down the radii n by thefirst narrowing condition (whether the number of votes obtained in thevote table 230 a is equal to or greater than the predetermined value,and whether which the number of votes obtained at the block in the votetable 230 b is equal to or greater than the predetermined value) toobtain the center points and the radii. Note that, as described above,the voting module 166 also registers, according to the numbers of votesobtained in the vote tables 230 a and 230 b, the corresponding points304 with the center point candidate list 310 at suitable timings.However, the registration with the center point candidate list 310 isperformed in the middle of the vote in which the final number of votesobtained is still unknown, and is not to determine the number of votesobtained at the time of the vote being finished. Thus, in this example,since the corresponding points 304 registered with the center pointcandidate list 310 are again compared uniformly with a largerpredetermined value than that at the time of registration, it ispossible to leave only appropriate corresponding points 304, whileexcluding other corresponding points 304 equivalent to noise.

Next, the traffic sign identifying module 168 derives, based on thecenter position and the radius n, a rectangular area of which one sidehas twice the length of the radius n and which is centering on thecenter position, as an occupying area. Note that, if the occupying areasoverlap (superimpose) with each other for any two corresponding points304, one area may become impossible to be recognized because of theother area. In such a case, if the traffic sign at the correspondingpoint 304 of the area that is impossible to be recognized is animportant traffic sign, such as a traffic sign that presents a speedlimit, a situation where such an important traffic sign is notrecognized may be caused. Therefore, if an occupying area at onecorresponding point 304 overlaps with another occupying area at theother corresponding point 304 in the screen (second narrowingcondition), the traffic sign identifying module 168 excludes a lessreliable one of the corresponding points 304 from the traffic sign, andleaves a more reliable traffic sign. Such a reliability of the trafficsign is calculated based on the comparison between the number of votesobtained in the two-dimensional vote table 230 a and the number of votesobtained in the three-dimensional vote table 230 b.

FIG. 16 is the flowchart illustrating one example of the traffic signidentifying processing. As illustrated in FIG. 16, the traffic signidentifying module 168 sequentially selects two candidates from thecandidates of the traffic sign (S330). Then, the traffic signidentifying module 168 determines whether the occupying areas of the twoselected candidates overlap with each other (S332). As a result, if theoccupying areas of the two candidates overlap (YES at S332), the trafficsign identifying module 168 determines whether both the numbers of votesobtained C1 and D1 in the vote tables 230 a and 230 b of one of thecandidates are greater than the number of votes obtained C2 and D2 inthe vote tables 230 a and 230 b of the other candidate (S334). As aresult, if both are greater, that is, if C1>C2 and D1>D2 (YES at S334),the traffic sign identifying module 168 excludes the other candidate(S336). If the occupying areas of the two candidates do not overlap (NOat S332), the processing transits to Step S344.

If other than C1>C2 and D1>D2 (NO at S334), the traffic sign identifyingmodule 168 determines whether both the numbers of votes obtained C1 andD1 in the vote tables 230 a and 230 b of one of the candidates are lessthan the number of votes obtained C2 and D2 in the vote tables 230 a and230 b of the other candidate (S338). As a result, if both are less, thatis, if C1<C2 and D1<D2 (YES at S338), the traffic sign identifyingmodule 168 excludes the one candidate (S340). Thus, if both the numbersof votes obtained in the vote tables 230 a and 230 b of the candidatesare greater, both the candidates with less numbers of votes obtained areexcluded, while leaving only greater numbers of votes obtained, becausethe reliability of the candidates with the greater numbers of votesobtained being a traffic sign is high.

If other than C1<C2 and D1<D2 (NO at S338), the traffic sign identifyingmodule 168 excludes either one of the candidates that is located lowerthan the other, based on the positions of one candidate and the othercandidate in the color image 126 (S342). Thus, since it cannot determineonly by the numbers of votes obtained if either one of the numbers ofvotes obtained in the vote tables 230 a and 230 b of the candidates isgreater and the other is less, only one candidate located higher isadopted and the other candidate located lower is excluded. This isbecause, if two traffic signs are disposed vertically at a higherposition and a lower position, respectively, one traffic sign presentinga speed limit, which is relatively important, is disposed above theother traffic sign.

When the two candidates thus selected overlap, the traffic signidentifying module 168 determines whether all the combinations of twocandidates to be selected have been finished after it determines thatthe one to be excluded (S344). As a result, if it has been finished (YESat S344), the traffic sign identifying module 168 ends the traffic signidentifying processing concerned, and if it has not been finished (NO atS344), the traffic sign identifying module 168 repeats the processingfrom Step S330. Thus, even if the occupying areas of the two candidatesof the traffic sign overlap, it is possible to appropriately narrow downto a reliable candidate.

Next, the traffic sign identifying module 168 determines whether thecandidate of the traffic sign narrowed down by the first and secondnarrowing conditions exceeds the candidate upper limit (here, 3), as thethird narrowing condition. Here, if the candidate upper limit isexceeded, the traffic sign identifying module 168 narrows down thecandidates to below the candidate upper limit, and does not performsubsequent processing for other candidates. Particularly, if thecandidate of the traffic sign exceeds the candidate upper limit, thetraffic sign identifying module 168 compares the horizontal distances xat the three-dimensional positions of all the candidates, and narrowsthe candidates of the candidate upper limit in order that the horizontaldistance x from the lane of the vehicle 1 is shorter. Thus, thecandidates near the lane of the vehicle 1 can be appropriatelyextracted, which are highly-possibly a traffic sign for the vehicle 1.

Next, the traffic sign correcting module 170 corrects the positionand/or the size of the traffic sign that is narrowed down to thecandidate upper limit or below the candidate upper limit. This isbecause the template matching is used for recognizing the content of thetraffic sign in this example, and the template matching is significantlyinfluenced on the recognition accuracy by positional offsets of theimages. Thus, the center position and the radius n derived by the votingmodule 166 are corrected, and the occupying area of the traffic sign isagain set. Therefore, the traffic sign correcting module 170 detects ared frame that exists in four horizontal and vertical directions fromthe center position of each candidate of the traffic sign, and correctsthe occupying area to form a rectangular area that is circumscribed bythe red frame (circumference part of the traffic sign). Particularly,the occupying area is corrected by the following procedures (1) to (7).

FIGS. 17A and 17B are diagrams illustrating processing of the trafficsign correcting module 170.

(1) First, the traffic sign correcting module 170 sets a rectangulararea as an occupying area 346, of which one side is twice the length ofthe radius n and has the center at the center position, as illustratedin FIG. 17A, and calculates a histogram of V-component of the entireoccupying area 346 (votes for the horizontal axis as the V-component).Then, the traffic sign correcting module 170 determines that thecandidate concerned is a red frame, if a difference between the maximumvalue and the minimum value of the histogram of V-component becomesequal to or greater than a predetermined value. If the differencebetween the maximum value and the minimum value of the histogram ofV-component is less than the predetermined value, the traffic signcorrecting module 170 determines that the candidate concerned is a blackframe, and replaces the histogram of V-component by a histogram of Ycomponent. Although the correction processing of the occupying area 346is described using the candidate with the red frame as an example, theprocessing is also applicable to a black frame.

(2) The traffic sign correcting module 170 derives a value ofV-component (threshold Vthr) that corresponds to a predetermined percent(e.g., 30%) from the highest interval in the histograms of theV-component, where the ratio of higher intervals and lower intervals inarea becomes 3:7, if the histogram is calculated by area). The thresholdVthr differs from the predetermined value of V-component used foridentifying the feature point. This is for setting an optimal thresholdfor each candidate. Note that, if the threshold Vthr becomes equal to orbelow the predetermined value (e.g., −5), the traffic sign correctingmodule 170 does not perform subsequent processing because the thresholdcan be considered to be an inappropriate threshold.

(3) As illustrated by arrows in FIG. 17B, the traffic sign correctingmodule 170 determines whether the V-component of each pixel is equal toor greater than the threshold Vthr from the center position of eachcandidate of the traffic sign, while shifting the detection pixel in thefour horizontal and vertical directions. Then, if the pixels of whichthe V-component becomes equal to or greater than the threshold Vthrcontinues for a predetermined number (e.g., 3 pixels), the firstdetected pixel among the pixels of which the V-component becomes equalto or greater than the threshold Vthr is selected as an inner edge 348of the red frame. Although the four horizontal and vertical directionsare described herein as the detecting directions, they may be any fourradial directions that intersect perpendicular to each other, withoutany limitation. Further, any other slanting directions, etc. withrespect to the four directions may also be added to improve thedetection accuracy.

(4) Next, the traffic sign correcting module 170 determines whether theposition of the inner edge 348 of the red frame is within apredetermined range where it should originally be located. Particularly,for example, in a case where the detection is performed in thehorizontally rightward direction, and assuming that the lateralcoordinate of the center is J, the x-coordinate at the horizontal rightend of the occupying area 346 is R, and the obtained coordinate of theinner edge 348 of the red frame is RE, the traffic sign correctingmodule 170 does not perform subsequent process because the coordinate REof the inner edge 348 of the red frame is an inappropriate value, if thefollowing Equations 3 is satisfied.RE<(R−J)×K+J  (Equation 3)

Here, K is a coefficient that takes any value of 0 to 1. For example, Kis set to 0.6 upon the horizontal detection, and is set to 0.5 in thevertical detection. Further, (R−J)×K is a lower limit in the radialdirection that the inner edge 348 can take (inner edge lower limit).This processing is a countermeasure for, for example, preventing thatthe inner edge 348 is erroneously taken due to the influence of theV-component, when the numerical value becomes orange in an electriclight display type traffic sign.

(5) Next, the traffic sign correcting module 170 again derives thecenter position and the radius n of the image based on the position ofeach inner edge 348, when any traffic sign is adapted in all fourhorizontal and vertical directions. Particularly, the center position ofthe inner edges 348 in the horizontal directions can be determined, ifthe inner edge 348 located at the left is LE and the inner edge 348located at the right is RE, by (LE+RE)/2, and the radius n can bedetermined by (RE−LE)/2. Further, the center position and the radius ncan be determined by similar processing for the vertical inner edge 348.Thus, the occupying area 346 identified by the center position of thetraffic sign is newly defined.

(6) Next, the traffic sign correcting module 170 compares, for thetraffic sign, the radius n before the correction with the radius n afterthe correction, and, if a ratio of the radii is deviated from apredetermined range (e.g., equal to or greater than 0.75 times and lessthan 1.5 times), the traffic sign correcting module 170 does not performsubsequent processing because the ratio is an incongruent value.

(7) Finally, the traffic sign correcting module 170 resizes theoccupying area 346 after the correction into a rectangular area ofpredetermined pixels in the horizontal directions×predetermined verticalpixels, and ends the correction processing concerned. Thus, it becomespossible to achieve the high recognition accuracy of the patternmatching by readjusting the center position and the radius of thetraffic sign. Note that general approaches, such as the nearest neighboralgorithm can be used as the resizing.

Traffic Sign Content Recognition Processing S204

FIG. 18 is a flowchart illustrating a flow of particular processing ofthe traffic sign content recognition processing S204. The traffic signcontent recognizing module 172 disperses, for the traffic sign correctedby the traffic sign correcting module 170, the luminance of the imagecorresponding to the occupying area 346 using the same image as thecorrection (S350), and determines whether this traffic sign is a trafficsign that presents a removal of a speed limit (S352). If the trafficsign is not a traffic sign that presents a removal of a speed limit, thetraffic sign content recognizing module 172 determines the traffic signto be a traffic sign that presents a speed limit and performs a verticalalignment of the traffic sign (S354), and then performs horizontalmatching (S356). Next, the traffic sign content recognizing module 172focuses on a predetermined part of the content of the traffic sign andperforms the template matching of the observing part (S358), and derivesan overall evaluation value and determines to which speed limit theobserving part corresponds (S360).

Meanwhile, as described above, there are an electric light display typeand a non-electric light display type of the traffic signs that arehandled as the targets in this example. The electric light display typeis higher in the luminance at the content of the traffic sign (e.g.,numerical value part) than the circumference, and the non-electric lightdisplay type is lower in the luminance at the content of the trafficsign (e.g., numerical value part) than the circumference.

In the traffic sign content recognition processing S204, the recognitionprocessing is performed assuming the possibility of both the indicationtypes, since either one of the indication types has not yet beengrasped. For example, the traffic sign content recognizing module 172performs a series of processing of Steps S350-S360 using the image ofthe traffic sign as it is that is corrected by the traffic signcorrecting module 170, and determines whether the content of the trafficsign is validly recognized (S362). As a result, if the traffic signcontent recognizing module 172 determines that the content of thetraffic sign is not validly recognized in any one of the processing ofSteps S350-S360 (NO at S362), it inverts the luminances of the trafficsign corrected by the traffic sign correcting module 170 (S364), andagain performs the series of processing of Steps S350-S360 for thetraffic sign that is inverted in the luminance (inverted traffic sign).

If the traffic sign content recognizing module 172 determines that thecontent of the traffic sign is validly recognized in any of theprocessing of Steps S350-S360 (YES at S362), it transits the processingto Step S366, without inverting the traffic sign or without recognizingthe content of the inverted traffic sign. Thus, either one of thetraffic sign corrected by the traffic sign correcting module 170 and theinverted traffic sign can be recognized, and it becomes possible toappropriately recognize the content of the traffic sign, regardless ofthe difference in the indication type, such as the electric lightdisplay type and the non-electric light display type.

Alternatively, if the traffic sign content recognizing module 172determines that the content of the traffic sign is not validlyrecognized in any one of processing of Steps S350-S360 before theinversion, it may interrupt the processing even during the middle ofrecognition processing, omit subsequent processing, and transit theprocessing to Step S362. Thus, unnecessary recognition processing can beavoided and the processing load can be reduced. Since the sameprocessing is applied to the image after the correction and the imagethat is inverted in the luminance, only the image after the correctionis described, and detailed description of the image that is inverted inthe luminance is omitted for convenience of explanation.

Here, although the image that is inverted in the luminance is processedafter processing of the image after the correction, this order may bereversed. For example, if a traffic sign that presents a speed limit islocated at a road shoulder, according to the vehicle exteriorenvironment, since the possibility that the traffic sign is anon-electric light display type is high, the image after the correctionis first processed, and if the traffic sign is located at a gate, sincethe possibility that the traffic sign is an electric light display typeis high, the image that is inverted in the luminance is first processed.Thus, since the image that is high in the possibility that theevaluation can be finished with a single loop of processing (StepsS350-S360) is first processed, the efficiency of the traffic signcontent recognition processing S204 can be improved.

If the content of the traffic sign can be validly recognized from theimage after the correction or the invert image that is inverted in theluminance (YES at S362), the traffic sign content recognizing module 172determines whether such processing of Steps S350-S364 is executed to allthe traffic signs corrected by the traffic sign correcting module 170(S366). As a result, if all traffic signs have not been finished (NO atS366), the traffic sign content recognizing module 172 repeats theprocessing from Step S350 until it finishes (YES at S366). Next, theprocessing of Steps S350-S360 is described in detail.

Luminance Dispersion Processing S350

The traffic sign content recognizing module 172 disperses the luminanceof each pixel over each occupying area 346 of the traffic signscorrected by the traffic sign correcting module 170. Thus, the image isconverted into an image pattern that can be recognized without dependingon the imaging state.

FIG. 19 is a diagram illustrating a recognition target area 370. First,the traffic sign content recognizing module 172 sets an area 370 towhich recognition processing in the rectangular area of thepredetermined pixels in the horizontal directions×predetermined verticalpixels is applied (hereinafter, referred to as “the recognition targetarea”). The recognition target area 370 is a rectangular area thatcontacts the inner edge 348 of the red frame in the occupying areas 346illustrating in FIG. 19, and the following processing is applied to therecognition target area 370.

Next, the traffic sign content recognizing module 172 disperses eachpixel of the recognition target area 370 to convert it into N-ary value.For example, if N=2, the luminance of each pixel has a value of either 0or 255. Note that N is a value of 2 or greater. In this example, N=5 inorder to reduce the effects to that pattern matching when thebinarization does not work well due to the effects of the thresholdsettings, etc. In a case of quinary (5-ary), the number of thresholds ofdispersion is four, and four predetermined percents (e.g., 20, 25, 35and 40%) from the highest interval in the histogram of luminance areselected for the thresholds. These predetermined percents can beselected independently and arbitrarily.

With this configuration, the content of the traffic sign isappropriately recognized, regardless of the difference in distributionof the luminance value. Moreover, since the recognition is based on thevalues of the higher intervals in the histogram of luminance,quinarization can be appropriately performed regardless of thedistribution state of the luminance in each recognition target area 370and, thus, normalization can also be achieved in addition to thedispersion.

Speed Limit Removal Determination Processing S352

Although the traffic sign content recognizing module 172 performs, forthe quinarized recognition target area 370, the recognition processingcorresponding to either one of the traffic sign that presents a speedlimit and the traffic sign that presents a removal of the speed limit,since the former case can make the processing load smaller, the trafficsign content recognizing module 172 first processes on the assumptionthat it is the former case, and if it is not the former case, thetraffic sign content recognizing module 172 then processes for thelatter case. Thus, it can be avoided to unnecessarily perform therecognition processing of the traffic sign that presents the speedlimit.

FIG. 20 is a diagram illustrating the traffic sign that presents aremoval of the speed limit. The traffic sign content recognizing module172 integrates luminances of a plurality of pixels corresponding to fourline segments L1, L2, L3, and L4 that cross in the recognition targetarea 370 illustrated in FIG. 20 and that have an angle of inclination(inclined) to obtain integrated luminance values S1, S2, S3, and S4 ofthe line segments L1, L2, L3, and L4, respectively. The angle ofinclination of the line segment is an angle that corresponds to theindication type of the traffic sign that presents a removal of the speedlimit. Here, the thresholds for determining the integrated luminancevalues S1, S2, S3, and S4 are TS1, TS2, TS3, and TS4, respectively. Notethat TS4 is comprised of two thresholds TS4 a and TS4 b having arelation of TS4 a<TS4 b. Alternatively, a relation of TS2=TS3 may alsobe adopted. The traffic sign content recognizing module 172 recognizesthe traffic sign concerned to be a traffic sign that presents a removalof the speed limit based on the following Equation 4 that quantifies thedifference in the intensity of the four line segments L1, L2, L3, andL4, when satisfying all of Equation 4.S1<TS1S2>TS2S3>TS3TS4a<S4<TS4b  (Equation 4)Here, since the deviations in the luminance due to the positional offsetand the brightness are corrected, the content of the traffic sign can berecognized by very simple processing like Equation 4 described above.

Vertical Alignment Processing S354

If the traffic sign is not determined to be a traffic sign that presentsa removal of the speed limit by the above-described processing, it isdetermined to be a traffic sign that presents the speed limit. Thetraffic sign content recognizing module 172 first performs a verticalalignment of a numerical area where the numerical values occupy withinthe recognition target area 370. This is because the recognition targetarea 370 may include a very small positional offset or may differ in thesize and the shape of the numerical value, and the distance between thenumerical values, etc. depending on country or installation style in thecountry.

FIG. 21 is a diagram illustrating the vertical alignment processing. Asillustrated in FIG. 21, the traffic sign content recognizing module 172horizontally integrates luminances of the pixels in the recognitiontarget area 370, develops the integrated luminance values in thevertical directions to generate a vertical luminance distribution 372.Note that, in the case of the traffic sign illustrated in FIG. 21, sincethe numeric part is lower in the luminance than other part around thenumeric part, the integrated luminance values are calculated afterinverting the luminances in the luminance distribution 372 of FIG. 21 inorder to extract the numeric part. Next, the traffic sign contentrecognizing module 172 calculates maximum values 374 of the integratedluminance values on both vertically upper and lower sides from thecenter part of the recognition target area 370, uses a predeterminedpercent (e.g., 25%) of each maximum value as a threshold to shift thedetection pixel vertically upward and downward from the center part,respectively, and if the pixels of that the integrated luminance valueis less than the threshold continue for a predetermined number (e.g., 2times), this location is selected as an upper end or a lower end of thenumerical area 376.

Next, the traffic sign content recognizing module 172 uses the upper endand the lower end of the numerical area 376 that are thus derived tonormalize the vertical size of the numerical area 376 by expanding orcontracting the vertical size. For example, if a distance between theupper end and the lower end of the numerical area 376 is HI, and avertical distance of a template is HT, the traffic sign contentrecognizing module 172 vertically multiplies the numerical area 376 byHT/HI times. Thus, the size of the numerical area 376 can be conformedto the vertical size of the template that is used for the matchingafterwards. Note that the correction is performed by the nearestneighbor algorithm.

Although noise is generated in the integrated luminance value, forexample, in the downward direction of FIG. 21 by such processing, theeffects can be eliminated and the normalized numerical area 376 can beextracted appropriately.

Horizontal Matching Processing S356

FIG. 22 is a diagram illustrating the template. The traffic sign contentrecognizing module 172 recognizes the content of the recognition targetarea 370 (i.e., the numerical value) that is aligned vertically. Thisrecognition is performed by matching with the templates prepared inadvance. As illustrated in FIG. 22, 13 kinds of templates, such as 10 to90 (two digits) and 100 to 130 (three digits), are prepared by a pitchof 10, for example.

FIG. 23 is a diagram illustrating horizontal matching processing. Thetraffic sign content recognizing module 172 vertically integrates thepixels in the vertically-normalized recognition target area 370 asillustrated in FIG. 23, it horizontally develops the integratedluminance values to generate a horizontal luminance distribution 380.Thus, the two-dimensional image is lowered to one dimension. Note that,similar to the vertical alignment, since the numeric part is low in theluminance, the integrated luminance value is calculated after invertingthe luminances in the luminance distribution 380 of FIG. 23 in order toextract the numeric part. Further, the vertical integrating range isonly a range from the upper end to the lower end of the numerical area376 derived by the vertical alignment. Therefore, an integration of theluminances of the unnecessary areas other than the vertical numericalarea can be avoided. The traffic sign content recognizing module 172carries out DP matching of the luminance distribution 380 in FIG. 23that is derived as described above with luminance distributions based onthe templates to calculate a correlation evaluation value with eachtemplate (here, the correlation is higher as the correlation evaluationvalue becomes lower).

Here, similar to the vertical alignment, since there are thedifferences, for example, in the size of the numerical value and thedifferences in the interval between the numerical values (i.e., the sizeof the gap), sufficient performance cannot be obtained if the templatesof a fixed size is used. Therefore, the DP matching in which horizontalexpansion and contraction are permitted is used. Although it istheoretically possible to perform the DP matching in two dimensions,since necessary throughput becomes significantly high, one-dimensionalDP matching is used in this example.

FIG. 24 is a chart illustrating the DP matching. The traffic signcontent recognizing module 172 performs the DP matching with, forexample, a template of “130,” and obtains the result as illustrated inFIG. 24. Here, a dashed line illustrates the luminance distribution 380of the recognition target area 370 before the DP matching, a solid lineillustrates a luminance distribution 382 of the recognition target area370 after the DP matching, and a dashed dotted line illustrates aluminance distribution 384 of the template. The DP matching is toperform matching while expanding and contracting the luminancedistribution 380 of the recognition target area 370 so that the size ofthe luminance distribution 380 becomes the same size as that of theluminance distribution 384 of the template. Therefore, as understoodfrom FIG. 24, the correlation of the luminance distribution 380 of therecognition target area 370 before expansion and contraction with theluminance distribution 384 of the template is low, but the correlationof the luminance distribution 382 of the recognition target area 370after expansion and contraction with the luminance distribution 384 ofthe template becomes high.

Note that, here, all the correlation evaluation values of the luminancedistribution 382 of the recognition target area 370 after expansion andcontraction with the luminance distribution 384 of the plurality oftemplates are calculated, regardless of the correlation evaluationvalues. Particularly, if the luminance distribution 382 of therecognition target area 370 after expansion and contraction is im, andthe numerical value (speed limit) of the template is T, the traffic signcontent recognizing module 172 sequentially derives DP(im, T) that is acorrelation evaluation value (sum of squares of difference) afterexpansion and contraction, from DP(im, 10) to DP(im, 130).

Note that subsequent processing is not performed for candidates that areclearly different from the templates. For example, the luminancedistribution 382 of the recognition target area 370 is “130,” and “10”to “90” of two digits differ in the digit count in the first place.Therefore, DP(im, 10) to DP(im, 90) corresponding to “10” to “90” arelow in the correlation. Accordingly, the following processing is omittedfor templates of that the value of DP(im, T) exceeds the threshold (lowcorrelation).

Observing Part Matching Processing S358

Here, although the correlation evaluation value DP(im, T) is calculatedregardless of the digit count of the numerical value, it is not wise toperform the matching with all the numerical values of the second digitand the third digit when the tendency of the change in numerical valueis known in advance like this example. This is because, for example, thefirst digit part of “0” is common for all the numerical values of “10”to “90,” and the first digit part of “0” and the third digit part of “1”are common for “100” to “130.” Therefore, since all the numerical valuesof the common part are identical, it is difficult to cause differencesin the correlation evaluation value if the matching is performed for allthe digits.

Therefore, the traffic sign content recognizing module 172 calculatesthe correlation evaluation values DP(im, T) as described above, andperforms the matching only at the second digits where a difference inthe shape of the numerical value is caused. Note that, since theluminance distribution 382 of the recognition target area 370 isexpanded and contracted horizontally, it must derive that part of theluminance distribution 382 of the recognition target area 370 isidentical to which part of the luminance distribution 384 of thetemplate. Therefore, the traffic sign content recognizing module 172derives horizontal coordinates DPR(im, T) that correspond to horizontalcoordinates TS(T) at the starting position of the second digit of thetemplate, and that correspond to the starting position of the seconddigit of the luminance distribution 382 of the recognition target area370. The horizontal coordinates can be calculated based on the historyof processing that links up the feature points in the DP matching.Particularly, the information on the combination of feature points(e.g., route) is stored in advance, and the horizontal coordinates arederived by calculating it backward. With this configuration, thehorizontal coordinates can be efficiently calculated using the result ofthe DP matching still in progress. Since particular procedures of suchDP matching have already been disclosed in various technicalliteratures, the detailed description thereof is omitted herein.

FIG. 25 is a diagram illustrating the matching processing of theobserving part. When the horizontal coordinates corresponding to thestarting position of the numerical area at the second digit can be foundas described above, the traffic sign content recognizing module 172performs simple template matching. Although any index may be used forthe template matching, the Sum of Absolute Difference (SAD) may be used,for example. A target range of the matching is, as illustrated in theexample of “130” of FIG. 25, part having a horizontal length of thenumerical value at the second digit of the template, based on thehorizontal coordinates DPR(im, T) corresponding to the starting positionof the luminance distribution 382 of the recognition target area 370,and the horizontal coordinates TS(T) corresponding to the startingposition at the second digit of the template. Here, since the horizontalalignment has already been performed, the processing load can besignificantly reduced, without necessity of processing, such as anoptimum value search by positional offset matching.

Note that, for example, since, the horizontal length (lateral width ofthe numerical value) differs between a two-digit numerical value and athree-digit numerical value, the result of matching may be affected bythe difference in the lateral width of the numerical value. Thus, thetraffic sign content recognizing module 172 multiplies a correlationevaluation value TM(im, T) of the DP matching at the second digit by anormalization coefficient defined in advance for each template,according to a ratio of the lateral width of the recognition target area370 of the numerical value with more digits and the lateral width of therecognition target area 370 of the numerical value with less digits. Forexample, if the ratio of the lateral width of the numerical value of twodigits against the numerical value of three digits is 3:2, the trafficsign content recognizing module 172 derives the correlation evaluationvalues TM(im, T) at the second digit of “100” to “130,” and thenmultiplies these values by 3/2 to replace TM(im, T) with the results ofthe multiplication. Thus, an appropriate evaluation can be performedregardless of the number of digits.

Evaluation Value Determination Processing S360

Next, the traffic sign content recognizing module 172 derives an overallevaluation value E(im, T) for each template by the following Equations 5and 6 using the correlation evaluation value DP(im, T) calculated foreach template and the correlation evaluation value TM(im, T) at thesecond digit.Overall evaluation value E(im, T)=DP(im, T)×TM(im, T)/F(im)   (Equation5)F(im)=max(min(TM(im, T)), th)   (Equation 6)Here, since the correlation of the entire numerical value is expressedby DP(im, T), a partial correlation at the second digit is expressed bycomparisons with other templates by using the same value of thecorrelation (i.e., TM(im, T)/F(im)). Here, although F(im) is a minimumvalue min(TM(im, T)) of the correlation evaluation value, since theoverall evaluation value E(im, T) may be diverged if the value ofmin(TM(im, T)) becomes too small, the predetermined value th is adoptedas F(im) when min(TM(im, T)) becomes less than the value th.

FIG. 26 is a diagram illustrating evaluation results. Here, a dashedline illustrates the correlation evaluation value DP(im, T), a dasheddotted line illustrates the correlation evaluation value at the seconddigit, TM(im, T), and a solid line illustrates the overall evaluationvalue E(im, T). As referring to FIG. 26, although the differencesbetween the original numerical value and other numerical values aresmall in the matching at all the digits of the numerical value for “100”to “130,” since the matching only at the second digit is added, it canbe understood that the overall evaluation value E(im, T) of the matchingwith the template of “130” that is the actual numerical value is theminimum (i.e., the correlation is the maximum). Note that, forfacilitating the description, FIG. 26 also illustrates templates forwhich the calculation is originally omitted, and illustrates templatesup to “150.”

Thus, since two steps of matching are carried out (matching at all thedigits of the numerical value and matching only at the second digit),the accuracy can be improved and the processing time can be reduced.

Traffic Sign Content Determining Processing S206

In the above, the content of the traffic sign is recognized. However, asdescribed above, it is not necessary to recognize a traffic sign at themoment of arriving the position where the traffic sign can be confirmedahead of the vehicle, but it is sufficient to recognize when or afterthe vehicle passes the traffic sign. Therefore, it is sufficient torecognize the traffic sign over a plurality of frames, and toconclusively determine the content of the traffic sign based on theinformation of the plurality of frames. Thus, the traffic sign contentdetermining module 174 integrates with time the contents of the trafficsign that is recognized in one frame to conclusively determine thetraffic sign.

Here, in order to conclusively determine the content of the trafficsign, four variables of a traffic sign integration point, a speed limitcandidate, a traffic sign non-detection time period, and a speed limitoutput are used. Here, the traffic sign integration point is preparedfor each of one or more candidates of the traffic sign, and indicates apoint according to the various evaluation values (E(im, T), DP(im, T),and TM(im, T)) in the traffic sign content recognition processing S204.The speed limit candidate indicates one candidate of a speed limit. Thetraffic sign non-detection time period indicates a continuous timeduring which any traffic sign is not detected. The speed limit output isused for a latch of the speed limit candidate. When the speed limitoutput is updated by the speed limit candidate, a value is held as thespeed limit output, and the value is informed to the vehicle operator orit is used as a control input of the vehicle control device 130 duringthe value being held.

The traffic sign content determining module 174 integrates the trafficsign integration points according to the conditions of the following (1)to (4) using the various evaluation values (E(im, T), DP(im, T), andTM(im, T)) derived by the traffic sign content recognizing module 172,by which the probability of the speed limit is evaluated.

-   (1) If E(im, T)<ETHR1 & DP(im, T)<DTHR1 & TM(im, T)<TTHR1, add +4    points;-   (2) If the condition (1) is not satisfied and if E(im, T)<ETHR2 &    DP(im, T)<DTHR2 & TM(im, T)<TTHR2, add +2 points, (here,    ETHR1<ETHR2, DTHR1<DTHR2, TTHR1<TTHR2);-   (3) Among the templates that satisfy the condition (1), if a    difference between the minimum value EM of E(im, T), and E(im, T) of    all other templates is equal to or greater than a predetermined    value, add +2 points to the template of the minimum value EM; and-   (4) Among the templates that satisfy the condition (1), if a    difference between the minimum value EM of E(im, T), and E(im, T) of    other one or more templates that are equal to or less than a    predetermined value (ETHR3), add +1 point to all the templates that    are equal to or less than the predetermined value.    Thus, fundamental points are added according to the conditions (1)    and (2), and points based on the comparison with other templates are    added according to the conditions (3) and (4).

For example, in the example of FIG. 26, suppose that the conditions (1)to (4) are ETHR1=80, ETHR2=100, ETHR3=5, DTHR1=80, DTHR2=100, TTHR1=100,and TTHR2=150, “120” and “130” are given +4 point based on the condition(1), “100” and “150” are given +2 points based on the condition (2), and“130” is given +2 point based on the condition (3). Putting thistogether, “130” is 6 points, “120” is 4 point, “100” and “150” are 2points, and other numerical values are 0 point. Note that, if recognizedas the traffic sign that presents the speed limit in the speed limitremoval determination processing, it is given uniformly 6 points.

The traffic sign content determining module 174 integrates the contentsof the traffic sign with time based on the traffic sign integrationpoints that are calculated as described above, and then performs a finaloutput, as follows.

FIG. 27 is a time chart illustrating a flow of result notification ofthe traffic sign. The traffic sign content determining module 174 alwaysintegrates the traffic sign integration points to all the candidates ofthe detected traffic sign for each frame. Then, the traffic sign contentdetermining module 174 compares the traffic sign integration point ofthe current frame with the traffic sign integration point of theprevious frame, and if the point does not change, the traffic signcontent determining module 174 increments the traffic sign non-detectiontime period because a traffic sign is not detected in the current frame,as illustrated in FIG. 27 (1). On the other hand, if the point changes,the traffic sign content determining module 174 resets the traffic signnon-detection time period to 0 because a traffic sign is detected in thecurrent frame, as illustrated in FIG. 27 (2).

Further, if the traffic sign is detected in the current frame, thetraffic sign content determining module 174 extracts the highest and thesecond highest traffic sign integration points of two traffic signcandidates from the plurality of traffic sign candidates, and speedlimits respectively recognized, as illustrated in FIG. 27 (3). Since itis conclusively determined at this time that the speed limit output isto be newly updated, the currently-held speed limit output is reset.Therefore, the notification of the speed limit is not performed. Thus,the unnecessary continuous notification of the previous speed limit canbe avoided.

In a case where the maximum value of the traffic sign integration pointexceeds the predetermined value (e.g., 8 points), the traffic signcontent determining module 174 updates the speed limit candidate withthe speed limit (e.g., 40) of the traffic sign of which the traffic signintegration point is the maximum value as illustrated in FIG. 27 (4)(i.e., it conclusively determines to be the output candidate) if thedifference with the second largest traffic sign integration point isequal to or greater than the predetermined value (e.g., 4 points). Thus,the candidate of the speed limit is extracted. If less than thepredetermined value, the speed limit candidate is updated to“Undetermined.” If the maximum value of the traffic sign integrationpoint is equal to or less than the predetermined value, the speed limitcandidate is not updated. Therefore, the speed limit candidate maintains“No Candidate.”

After there is no more change in the traffic sign integration point (thevehicle passed the traffic sign), the traffic sign content determiningmodule 174 determines whether the speed limit candidate exists asillustrated in FIG. 27 (5) if the traffic sign non-detection time periodelapses an output time period setting (e.g., 3 seconds). On the otherhand, if the speed limit candidate exists, the traffic sign contentdetermining module 174 updates the speed limit output by the speed limitcandidate (e.g., 40) to resume the notification of the speed limit.Thus, the noise input of the content of the traffic sign can beeliminated.

Next, after there is no more change in the traffic sign integrationpoint, the traffic sign content determining module 174 resets thetraffic sign integration point and the speed limit candidate in order toprepare for the next traffic sign as illustrated in FIG. 27 (6) if thetraffic sign non-detection time period elapses a reset time period(e.g., 5 seconds) that is longer than the output time period setting.Thus, a recognition of the new traffic sign is prepared.

Next, after there is no more change in the traffic sign integrationpoint, the traffic sign content determining module 174 resets the speedlimit output as illustrated in FIG. 27 (7) if the traffic signnon-detection time period elapses a notification maximum time period(e.g., 10 minutes). Thus, the unnecessary continuous notification of theprevious speed limit can be avoided.

If the vehicle 1 is determined to be during a left turn or a right turn(e.g., the absolute value of the steering angle exceeds 360°), thetraffic sign content determining module 174 resets all the traffic signintegration point, the speed limit candidate, and the speed limitoutput. This is because, when the vehicle 1 turns to the left or to theright, the road where the vehicle travels is changed, and the speedlimit of the previously-traveling road is no longer applicable.

With such a configuration, the traffic sign content determining module174 can notify the speed limit after 3 seconds from the passing of thetraffic sign, and maintain the speed limit until the lapse of 10minutes, the right turn or left turn, or other traffic signs aredetected. Moreover, the noise input of the content of the traffic signcan be eliminated, and the identification accuracy of the content of thetraffic sign can be improved.

In order to further improve the practicability, the following processingmay also be performed additionally. For example, when there are aplurality of lanes for the vehicle, the traffic signs disposed at a gatemay present different speed limits for each lane. In this example, sincethe number of the traffic signs to be recognized is limited to three orless, the traffic sign integration point exists for each of thedifferent traffic signs when the speed limits differ for each lane asdescribed above. If only the correct speed limits are integrated foreach traffic sign, the traffic sign integration point will be, forexample, 6 points, for any of the speed limits. In such a case, thespeed limit candidates will then be updated by “Undetermined” in thedetermination described above, regardless of the points areappropriately accumulated.

Thus, in this example, if a plurality of candidates of the traffic signof which the traffic sign integration points are significant valuesexist simultaneously in one frame, the traffic sign content determiningmodule 174 derives each horizontal distance x, and if the horizontaldistance x of any one of the traffic signs is less than a threshold(e.g., 3 m) and other traffic signs are equal to or greater than thethreshold, the traffic sign content determining module 174 adds 1 pointto the one traffic sign and subtracts 1 point from other traffic signs.Thus, the speed limit of the traffic sign that is nearest to the vehicle1 can be preferentially selected as the speed limit candidate.

Difference in Traffic Sign by Country

FIGS. 28A to 28D are diagrams illustrating indication types of thetraffic sign by country. As can be seen from the comparison between aGerman traffic sign that presents a speed limit in FIG. 28A and a Swisstraffic sign that presents a speed limit in FIG. 28B, the traffic signsthat present a speed limit may differ by country in the size and theshape of the numerical value, and the distance between the numericalvalues. Further, as can be seen from the comparison between a Germantraffic sign that presents a removal of a speed limit in FIG. 28C and aItalian traffic sign that presents a removal of a speed limit in FIG.28D, the traffic signs may differ in the angle of the slash.

Therefore, the traffic sign content determining module 174 determines tothat country's traffic sign the speed limit candidate belongs, inparallel to the integration of the contents of the traffic sign withtime for the conclusive determination as described above. Then, thetraffic sign content determining module 174 correctly grasps the countrywhere the vehicle is currently traveling, and appropriately recognizes,for example, the speed limit by using the templates of the countryconcerned.

FIG. 29 is a table illustrating the templates of the traffic signs.Country determination processing is fundamentally performed using thetemplates by country. Therefore, as illustrated in FIG. 29, thetemplates that are two-dimensionally arrayed by country (Country A toCountry Z) and by speed limit (“10” to “150”) are prepared. Note thatthe traffic signs that present a removal of the speed limit are not heldas the templates, but angle information of the slashes is held insteadof the templates. Further, information, such as the radii n for thetraffic sign detection processing S202, and the N-ary thresholds(predetermined percents) for the traffic sign content recognitionprocessing S204 are also held by country.

Here, if the vehicle exterior environment recognition system 100concerned is interlocked with a navigation system, the templates may beswitched according to the current country information obtained from thenavigation system; however, if the vehicle exterior environmentrecognition system 100 is not interlocked with the navigation system,the country determination is performed by the following procedures.

Note that the country determination processing is low in the requirementfor real-time processing. Therefore, when the image of the recognitiontarget area is once acquired, the image is temporarily held in atemporary image memory, and the country determination processing isperformed over two or more frames during an idle time after the vehicleexterior environment recognition processing for each frame is finished.Here, in order to determine the country, a total point by country thatis prepared for each one or more candidates of one country is used as avariable. The traffic sign content determining module 174 initializesthe temporary image memory area and the total point by country at apredetermined timing.

In the country determination processing, the traffic sign contentdetermining module 174 determines whether the country determinationprocessing has already been performed for the current frame. As aresult, if the country determination processing has already beenperformed, the traffic sign content determining module 174 continues thecurrent processing, and if the previous country determination processingis finished, the traffic sign content determining module 174 starts newcountry determination processing. As described above, since the countrydetermination processing is performed in the idle time, when apredetermined processing time period of the current frame is reached inthe middle of the current processing, the traffic sign contentdetermining module 174 suspends the current processing, and willcontinue the rest of the processing in the subsequent frame.

Next, the traffic sign content determining module 174 determines whetheronly one speed limit has obtained the traffic sign integration point(also including the case where the points have been obtained for aplurality of traffic signs) in one or more candidates of the trafficsign in the traffic sign content determining processing S206 describedabove for the current frame. As a result, if a traffic sign is notdetected, or if only the speed limit of one traffic sign has notobtained the traffic sign integration point, such as the points havebeen obtained for the plurality of speed limits for the plurality oftraffic signs, the traffic sign content determining module 174 ends theprocessing of the current frame and repeats the determination of thetraffic sign integration point in the subsequent frame.

Next, if only one speed limit has obtained the traffic sign integrationpoint, the traffic sign content determining module 174 determines therecognition result of the speed limit to be a speed limit V, and thenstores the image of the recognition target area 370 in the temporaryimage memory. Here, if the speed limit V has obtained the traffic signintegration point for all the plurality of traffic signs that arecandidates, the traffic sign content determining module 174 stores theimage of the recognition target area 370 of a candidate of which theoverall evaluation value E(im, V) is lowest (maximum correlation). Here,the image stored is the occupying area 346 that is normalized in therectangular area of the predetermined horizontal pixels×thepredetermined vertical pixels after the completion of the traffic signdetection processing S202.

Next, the traffic sign content determining module 174 performs thetraffic sign content recognition processing S204 for the templates ofthe speed limit V of each country based on the image of the occupyingarea stored in the image memory. That is, the traffic sign contentrecognition processing S204 described above uses templates of everyspeed limit T of one country as illustrated by a dashed line in thetable of FIG. 29; however, the templates of every country of one speedlimit V are used in this example as illustrated by a solid line in thetable of FIG. 29. Therefore, if an identifier of the country is CN and aspeed limit is V, the overall evaluation value E(CN, V) is derivedinstead of the overall evaluation value E(im, T). The evaluation valuesE(CN, V) for all the countries for which the templates are prepared canbe acquired in this processing.

Note that weighting is varied in the evaluation value E(CN, V) betweenthe currently-recognized country and other countries in this example.For example, the traffic sign content determining module 174 multipliesonly the evaluation value E(CN, V) of the currently-recognized countryby a weighting coefficient (e.g., 0.8 that is equal to or less than 1).This is for relatively lowering the evaluation value E(CN, V) of thecurrently-recognized country (increasing the correlation) and avoidinghunching from causing in the result of the country determination.Alternatively, if a country adjacent to the currently-recognized countrycan be grasped, the weighting coefficient (e.g., 0.95 that is equal toor greater than 0.8, and equal to or less than 1) may also be multipliedfor the adjacent country.

The traffic sign content determining module 174 compares the evaluationvalues E(CN, V) for all the countries thus derived to derive a minimumvalue ECM. If differences between the minimum value ECM and theevaluation values E(CN, V) of all other templates are equal to orgreater than a predetermined value, +1 is added to the correspondingtotal points by country of the template of the minimum value ECM.

Next, the traffic sign content determining module 174 compares themaximum value of the total points by country with all other total pointsby country, and if the differences are equal to or greater than apredetermined value (e.g., 30), the traffic sign content determiningmodule 174 determines whether the country with the maximum value isidentical to the currently-recognized country. As a result, if thecountry with the maximum value is identical to the currently-recognizedcountry, the traffic sign content determining module 174 multiplies allthe total points by country by ½ to lower all the traffic signintegration points for fair judgment. If the country with the maximumvalue differs from the currently-recognized country, the traffic signcontent determining module 174 determines that the country where thevehicle is traveling has been changed, updates the currently-recognizedcountry by the country with the maximum value, and initializes thetemporary image memory area and the total points by country. If thedifference between the maximum value of the total points by country andall other total points by country are less than a predetermined value,the traffic sign content determining module 174 initializes thetemporary image memory area, and repeats the determination of thetraffic sign integration point concerned in the subsequent frame.

Thus, the identification accuracy of the content of the traffic sign canbe improved by appropriately determining the currently-travelingcountry. Further, the processing load can be lowered by performing thecountry determination processing described above in the background ofthe recognition processing of the traffic sign of one country.

As described above, the vehicle exterior environment recognition device120 of this example is possible to improve the recognition accuracy ofthe content of the traffic sign, while reducing the processing load.

The vehicle exterior environment recognition device 120 may be providedas one or more computer-readable programs that can function one or morecomputers as the vehicle exterior environment recognition device 120, ormay be provided as one ore more storage media that record thecomputer-readable program(s), such as one ore more flexible disks,magneto-optic discs, ROMs, CDs, DVDs, and BDs. The term “program” asused herein refers to a data set that is described in any languageand/or any describing method.

Although the suitable example of the present disclosure is describedabove with reference to the accompanying drawings, it cannot beoveremphasized that the present disclosure is not limited to thisexample. It is apparent to a person skilled in the art that variouskinds of changes and/or modifications are possible without departingfrom the scope of the appended claims, and it should be understood thatthose changes and/or modifications naturally belong to the technicalscope of the present disclosure.

Note that the processes of the vehicle exterior environment recognitionprocessing described herein are not necessarily processed in the orderindicated in the flowcharts, and they may be parallelly processed or maybe processed by subroutine(s).

The present disclosure can be used for the vehicle exterior environmentrecognition device that recognizes the content of the traffic signinstalled on the road.

The invention claimed is:
 1. A vehicle exterior environment recognitiondevice, comprising: an image acquiring module that acquires an image; atraffic sign identifying module that identifies a circle of apredetermined radius centering on any one of pixels in the image as atraffic sign; a traffic sign content recognizing module that recognizescontent of the identified traffic sign; and a traffic sign contentdetermining module that uses templates for one certain country thatrespectively correspond to contents of traffic signs and determines onecontent that has a highest correlation with the content of theidentified traffic sign, and uses a template for each of a plurality ofcountries having a highest correlation with the content of theidentified traffic sign, and conclusively determines one country thathas a highest correlation with the content of the identified trafficsign.
 2. A vehicle exterior environment recognition device, comprising:an image acquiring module that acquires an image; a traffic signidentifying module that identifies a circle of a predetermined radiuscentering on any one of pixels in the image as a traffic sign; a trafficsign content recognizing module that recognizes content of theidentified traffic sign; and a traffic sign content determining modulethat uses templates for one certain country that respectively correspondto contents of traffic signs and determines one content that has ahighest correlation with the content of the identified traffic sign anduses a template for each of a plurality of countries having a highestcorrelation with the content of the identified traffic sign, andconclusively determines one country that has a highest correlation withthe content of the identified traffic sign, wherein the traffic signcontent determining module applies weighting to each of acurrently-recognized country and a country adjacent to thecurrently-recognized country so that the currently-recognized countryand the country adjacent to the currently-recognized country are easilyselected.
 3. The vehicle exterior environment recognition device ofclaim 1, wherein the traffic sign content determining module storesimages that are determined to be the identified traffic sign in an imagememory, and uses a template corresponding to the content of theidentified traffic sign for each of the plurality of countries todetermine conclusively determines the one country which has the highestcorrelation, during an idle time of the processing that uses thetemplates for the one certain country which respectively correspond tothe contents of the traffic signs and determines the one content whichhas the highest correlation with the content of the identified trafficsign.
 4. The vehicle exterior environment recognition device of claim 2,wherein the traffic sign content determining module stores images thatare determined to be the identified traffic sign in an image memory, anduses a template corresponding to the content of the identified trafficsign for each of the plurality of countries to determine conclusivelydetermines the one country which has the highest correlation, during anidle time of the processing that uses the templates for the one certaincountry which respectively correspond to the contents of the trafficsigns and determines the one content which has the highest correlationwith the content of the identified traffic sign.